Question order effects: how robust are survey measures on political solidarities with reference to Germany and Europe?

ABSTRACT

The measurement of political solidarities and related concepts is an important endeavor in numerous scientific disciplines, such as political and social science research. European surveys, such as the Eurobarometer, frequently measure these concepts for people’s home country and Europe raising questions with respect to the order of precedence. Research has shown that the order of asking questions can have a profound impact on answer behavior compromising data quality. In this study, we therefore investigated the occurrence of question order effects in a German-European context using two questions on political solidarities. For this purpose, we conducted an experiment in a German online panel (N = 874) and analyzed response behavior and effort in terms of response times. In contrast to previous research, we found no empirical evidence for question order effects impacting people’s responses. Even though there were no response time differences between the question order conditions, the first question always took longer to respond to than the remaining one. Overall, our findings indicate the robustness of questions on political solidarities against question order effects. One potential explanation is that people have comparatively strong (or crystallized) attitudes when it comes to political solidarities.

KEYWORDS:

• Response behavior
• data quality
• non-probability panel
• survey question order
• web survey

Acknowledgments

The authors are grateful to Joshua Claaßen (DZHW) for his great support during data collection and manuscript writing.

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes

1. Approx. 2% (n = 15) of the respondents selected the response option ‘divers.’ The remaining respondents selected the response option ‘male.’

Additional information

Funding

The authors are grateful to the SoSci Panel (www.soscipanel.de) – in particular, to Christina Peter (University of Klagenfurt) – for conducting the study free of charge. Parts of the study were supported by a Consolidator Grant of the European Research Council (ERC) that was awarded to Achim Goerres (POLITSOLID; Grant number: 864818).

Notes on contributors

Jan Karem Höhne

Jan Karem Höhne is junior professor at the University of Hannover in association with the German Centre for Higher Education Research and Science Studies (DZHW) and research fellow at the Research and Expertise Centre for Survey Methodology (RECSM). His research focuses on computational survey and social science.

Achim Goerres

Achim Goerres is a full professor of empirical political science at the Department of Political Science at the University of Duisburg-Essen and Principal Investigator of the POLITSOLID project funded by the European Research Council. His research interests lie at the intersection of political behaviour and welfare state research with a focus on ageing, immigration and solidarities.