Fear of punishment as a driver of survey misreporting and item non-response in Russia and its neighbors

ABSTRACT

Following its February 2022 invasion of Ukraine, the Russian government sharply broadened what actions were illegal and raised the level of punishment. Many more topics of interest to survey researchers became politically sensitive. Questions about these topics may generate high levels of misleading responses and question-specific (item) non-responses, both of which introduce biases that undermine inference. We use survey data from 2015 and 2018 in Russia and neighboring countries to illustrate how these two problems were already issues prior to the invasion, especially for questions that invoked potential punishment by the state. In a climate of heightened state punishment, it becomes even more important to address misresponse and item non-response when interpreting survey data. We argue that, in addition to employing list experiments regularly and taking advantage of recent innovations in their design, scholars must develop ways to reduce item non-response and model how it biases estimates of interest.

KEYWORDS:

• Russian public opinion
• survey research
• list experiments
• item non-response

Acknowledgments

These analyses were first presented at the European Conference on Political Research. We are grateful to Ilona Wysmulek for helpful comments. We also thank Timothy Frye and an anonymous referee for this journal for helpful suggestions.

Supplemental data

Supplemental data for this article can be accessed online at https://doi.org/10.1080/1060586X.2022.2150490.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

1. Note that we are not addressing “unit non-response” – when people refuse to be interviewed at all or stop the interview entirely once a sensitive topic has been broached. Unit response can lead to bad inferences because the sample is not fully representative. On this issue in comparative research, see James and Stoop (2018). As Rosenfeld (2023) shows in her Figure 1 in this issue, the Levada survey firm found no significant increase in unit non-response following the March 2022 invasion. See also her discussion on the challenges of achieving adequate response rates.

2. As Rosenfeld (2023) notes in this issue, list experiments can give rise to their own problems and will be most valuable when combined with other indirect methods. Note also that although a type of survey experiment, list experiments are not a way to identify a relationship using random control, the goal of the studies cited in Libman’s (2003) contribution to this issue. They are a diagnostic tool from which one can estimate the degree of misresponse to a different question.

3. Agerberg (2022) finds evidence of political sensitivity in response to questions about corruption from a survey experiment in Romania.

4. Respondents were read and agreed to a statement of informed consent. Both sets of surveys were approved by the University of Iowa Institutional Review Board 02, approval numbers 2014-07750 and 2018-02824.

5. We chose the non-sensitive items so that it would be rare for anyone to select either all or none of the non-sensitive items or to respond differently to the sensitive item because of the specific non-sensitive items on the list. These are referred to as a ceiling effect, a floor effect, and a design effect, respectively. We conducted analyses developed by Blair and Imai (2012) and found no meaningful floor, ceiling, or design effects, with one exception, which we discuss below. The full wording of the list experiment questions is provided in the online Appendix.

6. The averages for the corruption variable exclude those who are missing because they indicated no contact with officials in that sector.

7. But see Ahlquist’s (2018) argument that they are less useful for measuring rare events and behaviors.

Additional information

Funding

This work was supported in part by the US Army Research Office/Army Research Laboratory under grant number W911NF-14-1-0541 and by the US Department of Defense Minerva Initiative, study #W911NF-18-1-0078. The views expressed are those of the authors and should not be attributed to the Army Research Office/Army Research Laboratory or the US Department of Defense Minerva Initiative.