Metrics of News Audience Polarization: Same or Different?

ABSTRACT

Although media and communication scholars have suggested various analytical methods for measuring and comparing news audience polarization across countries, we lack a systematic assessment of the metrics produced by these techniques. Using survey data from the 2016 Reuters Institute Digital News Report on news use in 26 countries, we address this gap through a resampling simulation experiment. Our simulation revealed a strong impact of analytical choices, which invited disparate interpretations in terms of how polarized news audiences are, how strongly audience polarization structurally varies between news environments, and how news audience polarization is distributed cross-nationally. Alternative choices led to profound differences in the compatibility, consistency, and validity of the empirical news audience polarization estimates. We conclude from these results that a more precise methodological understanding of news audience polarization metrics informs our capability to draw meaningful inferences from empirical work.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

¹ While, more technically speaking, the lower-bound of the range of the empirically possible coefficient-of-determination values is determined by the bivariate correlation of ideology with the use of the news outlet whose use is most strongly correlated with ideology, it is also important to bear in mind that bivariate correlations are strongly subject to range restrictions by themselves. That is, the range of empirically possible correlation values is usually much narrower than the theoretical range, r = [−1; 1], unless the variables have identical marginal distributions. This is our case means that even when the use of an outlet is exclusively restricted to conservatives (or liberals), its correlation with ideology will not only be very weak if the outlet has a small reach. The correlation will also prototypically be smaller compared to outlets whose audience is less exclusively restricted to liberals (or conservatives) but which have a larger reach.

² Because, for instance, the standard deviation is essentially the average difference of the individual outlets’ leaning scores from the overall mean leaning in a country, it is generally more difficult to produce high news audience polarization scores with the spectrum approach than with the coefficient-of-determination approach. High spectrum scores presuppose that the use of a larger number of news outlets is limited to either liberals or conservatives. The spectrum approach also assigns weight to the individual news outlets that is directly proportional to the number of their users. In this respect, it is more closely related with the networked duplication-count approach than with the networked backbone approach, as introduced in the next section.

³ On more technical grounds, such agreement is also called covariance or rank-order stability/consistency and, with continuous measures, commonly calculated by their Pearson correlations in methodological research. This is done to acknowledge that neighboring rank places may correspond with either relatively small or large substantive differences among the respective pairs of individual countries, recipients, etc., or, more generally speaking, units of analysis (e.g., Goldman et al., 2013; Scharkow, 2019; Wonneberger & Irazoqui, 2017).

⁴ The commonalities and relative differences between the analytical approaches were robust against both smaller respondent samples and lower thresholds for including news outlets in the media samples with one slight exception: the coefficient-of-determination approach produced, though highly, somewhat less reliable estimates compared to the duplication-count and spectrum approaches when setting the threshold lower for including outlets in the media samples.

Additional information

Funding

The author(s) reported there is no funding associated with the work featured in this article.

Notes on contributors

Frank Mangold

Frank Mangold is a post-doctoral researcher in the Department of Computational Social Science at GESIS—Leibniz Institute for the Social Sciences. His research interests are audience structures and media repertoires, opinion leadership, and quantitative methods, especially data collection and statistical modeling.

Michael Scharkow

Michael Scharkow is professor for Computational Communication Science at the Department of Communication, Johannes Gutenberg University Mainz. His main research interests are online communication, media use and effects, and quantitative methods, especially data collection and statistical modeling.