Do surveys change behaviour? Insights from digital trace data

ABSTRACT

Surveys continue to be a popular way of collecting data in the social sciences. But, despite their popularity they have a number of limitations including the possibility of changing the behaviour of respondents. Such mere-measure or question-behaviour effects can compromise the external validity of social data. In this article, we use digital trace data collected from PCs and mobile devices to investigate the effects of surveys on news and politics consumption. Using a non-probability panel of respondents in Germany we combine the digital trace data with that from three online surveys regarding the federal election. In contrast to our expectation, the participation in the survey does not influence online news and politics media consumption. Furthermore, we find weak evidence that respondents with previous high media consumption are less likely to be influenced by doing the survey compared to those with low media consumption.

KEYWORDS:

• Question-behaviour effects
• survey data
• digital trace data
• elections
• conditioning

Acknowledgments

This work was supported by the European Union’s Horizon 2020 research and innovation programme under grant agreement No 730998, “InGRID-2 – Integrating Research Infrastructure for European expertise on Inclusive Growth from data to policy”; German Research Foundation (DFG) under grant agreement No 139943784, “Collaborative Research Center SFB 884 Political Economy of Reforms (Project A8).”

Disclosure statement

No potential conflict of interest was reported by the authors.

Supplementary material

Supplemental data for this article can be accessed here.

Notes

1. The prevalence of potentially sensitive behaviours (e.g. visits to adult content websites and illegal streaming platforms) suggests that users did not make use of the possibility top turn off tracking very often.

2. At the beginning of each of the surveys, participants were asked whether they were eligible to vote in the upcoming (waves 1 and 2)/past (wave 3) general election.

3. Table A2 in the appendix highlights the selection within analysis sample (i.e. having digital trace data at least once) and the participation rate by wave (where the non-respondents is the reference group).

4. How would you rate your interest in politics? Please use the following scale where 0 means ‘no interest at all’ and 10 means ‘very high interest.’

5. We also tried to model the number of surveys the respondents missed (i.e. one or two surveys) but the effect was not significant. Results are not shown.

Additional information

Funding

This work was supported by the European Union’s Horizon 2020 research and innovation programme under grant agreement No 730998, “InGRID-2 – Integrating Research Infrastructure for European expertise on Inclusive Growth from data to policy”; German Research Foundation (DFG) under grant agreement No 139943784, “Collaborative Research Center SFB 884 Political Economy of Reforms (Project A8).

Notes on contributors

Alexandru Cernat

Alexandru Cernat is a lecturer in the social statistics department at the University of Manchester. He has a PhD in survey methodology from the University of Essex and was a post-doc at the National Centre for Research Methods and the Cathie Marsh Institute. His research and teaching focus on: survey methodology, longitudinal data, measurement error, latent variable modelling, new forms of data and missing data. You can find out more about him and his research at: www.alexcernat.com

Florian Keusch

Florian Keusch is Professor of Statistics and Methodology (interim) at the University of Mannheim, Germany, and Adjunct Assistant Professor at the Joint Program in Survey Methodology (JPSM), University of Maryland. His research focuses on nonresponse and measurement error in (mobile) web surveys and digital trace data collection. You can find out more about him and his research at: https://floriankeusch.weebly.com/