Assessing (In)accuracy and Biases in Self-reported Measures of Exposure to Disagreement: Evidence from Linkage Analysis Using Digital Trace Data

ABSTRACT

Citizen’s exposure to disagreement – whether intentional or incidental – is a central concept in communication research, yet the precise degree to which citizens are exposed to opposing views online and the antecedents to this phenomenon continue to be debated. Despite the theoretical importance of this question, empirical assessments of cross-cutting exposure, especially those involving online settings, are largely based on individuals’ perception of their own behavior. Therefore, we know little regarding response bias in self-reports of cross-cutting exposure online. Combining digital trace data with a panel survey, we observe overreporting of self-reported online cross-cutting exposure. We then demonstrate that self-reported exposure to disagreement is retrospectively conditioned by the perception of the opinion climate in a given context. Finally, using Monte Carlo simulations, we examine the consequences of relying on (potentially imperfect) self-reported measures.

Acknowledgments

We acknowledge that the original data collection of this study has been funded by National Research Foundation of Korea, under the grant no. NRF-2013S1A3A2054988, awarded to Dr. Jong Hyuk Lee (School of Journalism & Mass Communication, Kyung Hee University, South Korea) and Dr. Yun Jung Choi (Division of Communication and Media Studies, Ewha Womans University, South Korea). We would like to thank Dr. Jong Hyuk Lee and Dr. Yun Jung Choi for making original data available for this paper. We also thank Dr. Jakob-Moritz Eberl and Dr. Loes Aaldering for their helpful comments on earlier drafts of the manuscript.

Disclosure Statement

No potential conflict of interest was reported by the author(s).

Supplemental data

Supplemental data for this article can be accessed on the publisher’s website.

Notes

1. Exposure to disagreement also goes by several other names (including “cross-cutting exposure,” “heterogeneous discussion,” and “discussion diversity”). Here, we use the terms “exposure to disagreement” and “cross-cutting exposure” largely interchangeably.

2. Studies using deliberative polls (e.g., Wojcieszak & Price, 2010, 2012) or experimentally-induced political discussions (e.g., Settle & Carlson, 2019) typically bypass this issue by using pre-determined contentious political topics such as gun control, abortion, or immigration issues.

3. The main page, displayed once participants log in to the website, carried the titles of the threads (the latest one at the top) with the user ID of the poster, click-view counts, and the counts of comments (if any). A separate minor section on the main page carried study-related information (e.g., reminder about surveys, announcements to encourage forum participation, etc.).

4. The results remained unchanged when we excluded independent categories (e.g., supporting a third-party candidate or being undecided: 6.39%, N = 22, only at the first wave) from the analysis.

5. It is worth noting here that the described measurement resembles what is called a summary network measure, which relies on summary questions on one’s frequency of political discussion with specific characteristics (here, perceived partisanship; see Hutchens et al., 2018, for a detailed discussion). While this approach is dominant in the political communication literature, it crucially differs from another prominent approach that solicits more specific, named discussants (i.e., name-generator approach). We return to this point in the discussion section.

6. The 95% CIs reported here are based on a nonparametric resampling-based permutation test (N = 20,000), where we randomly reshuffle the self-reported measure and the behavioral measure, derive the null distributions of differences, and compare this null distribution with the observed difference. See online appendix for more details.

7. We used the 90% highest posterior density intervals (HDI) for reporting credible intervals, as recommended by Gelman and Carlin (2014). To estimate models, we used regularizing priors (i.e., weakly informative priors: Gelman et al., 2017), with 32,000 posterior draws (4,000 iterations of eight chains each), using Hamiltonian Monte Carlo with the no-U-turn sampler as described in Hoffman and Gelman (2014). In the online appendix, we also report OLS regression models with nonparametric bootstrapping (N = 10,000) with the traditional 95% (instead of 90%) percentile CIs.

8. Our analysis assumes that participants’ perceptions and actual behaviors are under the different influences of social desirability, motivations and abilities, and public opinion perception, leading to systematic discrepancies between perceptions and actual behavior. We present additional analysis to directly validate this assumption in the online appendix (see section 5, “Sensitivity Analysis”).

Additional information

Funding

This work was supported by the National Research Foundation of Korea [NRF-2013S1A3A2054988].

Notes on contributors

Hyunjin Song

Hyunjin Song (PhD, The Ohio State University) is an assistant professor in the Department of Communication at Yonsei University, Seoul, South Korea. His research areas include the influence of interpersonal discussion on political engagement and statistical modeling of social networks.

Jaeho Cho

Jaeho Cho (PhD, University of Wisconsin-Madison) is a professor of Communication in the Department of communication at the University of California, Davis. His research concerns the influence of mass media and communication technologies on political decision-making and behavior.