https://orcid.org/0000-0002-1540-8004
![Sample our Humanities journals, sign in here to start your FREE access for 14 days]({"type":"image" , "src" : "/sda/5939/TOC-Subject-Sample-2021-Humanities.jpg"})
ABSTRACT
This paper assesses the performance of three commonly used types of questions – open-ended, check-all and forced choice – for capturing retrospective online news exposure, combining both survey and web-tracking data. It examines the performance of these different survey questions considering both systematic and random error in two unexplored non-US contexts: Spain and the UK. Results show that the check-all question produces on average the most accurate – i.e. less biased – estimates of observed exposure. Some motivational and cognitive factors underlying bias in self-reports are explored. Findings reveal that the characteristics of outlets are associated with systematic error. Finally, we find that media systems matter for accuracy – where media fragmentation is high (Spain), accuracy is low across all questions; where it is low (UK), accuracy is high across all questions. In the final section, we highlight the methodological and theoretical contributions of our study and provide some recommendations.
Acknowledgments
The authors are grateful to the Charlemagne Prize Academy Research Fellowship for funding support, and to YouYou Wu for her valuable assistance during the data pre-processing stage
Disclosure Statement
No potential conflict of interest was reported by the author(s).
Supplementary Material
Supplemental data for this article can be accessed on the publisher’s website at https://doi.org/10.1080/10584609.2022.2090038
Notes
1. See the evolution of time spent in the five most visited outlets in the months surrounding the outbreak of the pandemic for each country in Figure A1 in the Online Supplementary Material (p. 2), which can be found here: https://www.dropbox.com/s/zdpn4p4f1cg6pxx/Online%20Suplementary%20Material_R1%20.docx?dl=0.
2. Fragmentation here is not defined from an audience-centric approach (Fletcher & Nielsen, Citation2017; Ksiazek, Citation2011; Majó-Vázquez et al., Citation2019) but from a media perspective. At the media level, fragmentation takes on different values as a function of (i) the number of available sources and (ii) the distribution of the audience across these sources (Majó-Vázquez, Citation2017). The larger the set of available sources and the more widely distributed the audience is across these sources, the higher media fragmentation. Conversely, the smaller the number of available outlets and the more concentrated the audience is around a few outlets, the lower media fragmented.
3. We replicated the analyses on two datasets. To create the first dataset, we filtered in individuals who visited outlets only from their mobile devices during the whole study period. To create the second dataset, we filtered in visits from mobile devices. This dataset, thus, includes individuals using both devices but keeps only the visits from mobile devices. A more detailed explanation of these datasets is offered in pages 19 and 24 of the Online Supplementary Material. To make datasets comparable to the UK dataset and avoid underestimating overreporting, in both datasets we included individuals that were inactive, who are more likely to overreport (see analyses in the Online Supplementary Material, pp. 19–29; see, also Prior, Citation2009a).
4. Following Jürgens et al. (Citation2020), we checked for differential sampling bias by device. Individuals making use of their mobiles for news visits differ from those not using this device on college levels and ideology. On average, mobile news users are less educated (college education is 7% lower among this group) and less conservative (conservatism decreases 0.02 among this group). In most other relevant sociodemographic and attitudinal variables (age, gender, political interest, news interest….) mobile news users are indistinguishable from non-mobile users.
5. We also computed several audience-based measures with our observed and reported data following previous studies (Flaxman et al., Citation2016; Fletcher et al., Citation2019; Cardenal et al., Citation2019), but found several inconsistencies, which we attributed to the unconventional period of our analysis. To cross-validate our slant measure, we compared our results with an external survey, the Digital News Report. We found that a few results differ. But based on our knowledge of the media systems that we studied, we believe that our measure is consistent.
Additional information
Notes on contributors
Ana S. Cardenal
Ana S. Cardenal is a Professor of Political Science at the Universitat Oberta de Catalunya (UOC) and a Lecturer at the Universitat Pompeu Fabra (UPF). She has published two books in the field of Comparative Politics and has worked as a consultant for the Inter-American Development Bank (IDB). Her current research interests focus on digital and social media, online news audiences, preferences and opinion formation, and the use of computational methods in the social sciences.
María Victoria-Mas
María Victoria-Mas* is an Assistant Professor at the Universitat Internacional de la Rioja (Spain) and a research fellow in the Red Leonardo (Fundación BBVA, Spain). Between 2019-2020, she was a research fellow in the European Charlemagne Prize Academy (Aachen, Germany). Between 2018-2019, she was a visiting scholar at the Center for Media Engagement (Moody College of Communication, University of Texas at Austin, USA). Her main research areas are: news media management and media branding; news consumption and misinformation.
Silvia Majó-Vázquez
Silvia Majó-Vázquez is research fellow at the Reuters Institute for the Study of Journalism, University of Oxford, UK. Her research interests focus on online news audiences, digital and social media, social network analysis, and the use of computational methods in the social sciences. Her work has been published in Proceeding of the National Academy of Science, Political Communication, Journal of Computer Mediated Communication and Journal of Communication among others.
Iván Lacasa-Mas
Ivan Lacasa-Mas (Ph.D., Universitat Autònoma de Barcelona) is an Associate Professor at the Universitat Internacional de Catalunya in Barcelona and a visiting scholar at the Center for Media Engagement and at the Digital Media Research Program (University of Texas at Austin). His interests are focused on news media management and user engagement, on misinformation, media credibility and media literacy, and on the impact of the changing media environment on journalism, politics and culture.