The Unified Framework of Media Diversity: A Systematic Literature Review

Abstract

How to study media diversity has become a major concern in today’s media landscape. Many expect that algorithmic filtering and a shift of audiences from legacy media to new intermediaries decrease the diversity of news diets, leading to fragmented societies, polarization and spread of misinformation. Different fields, from journalism research to law and computer science, are involved in the study of media diversity. They operate, however, with vastly different vocabularies, frameworks, and measurements. To overcome this fragmentation, this study provides an extensive overview of conceptualizations and operationalizations of media diversity in different fields using a systematic literature review (1999–2018). This showed a lack of theorizing and linking of conceptual with empirical work in media diversity research. Based on this, we develop a framework on how to move forward: Regarding conceptualization, we call for focusing on different places in the journalistic information chain instead of the classical exposure-supply distinction. Methodologically, automated approaches (e.g., analyzing digital traces) and qualitative approaches (e.g., capturing perceptions of diversity) should receive more attention. For analysis, matters of balance and disparity need to be stressed more, especially discussing possible limits to diversity. Overall, research into media diversity thus needs to be addressed in interdisciplinary collaboration.

Keywords:

• Media diversity
• systematic review
• interdisciplinary work
• pluralism

Media diversity has always held a prominent place on the research agenda in communication research and journalism studies. But especially in the last few years, concerns about the diversity of media diets have taken the center stage of a lively academic and societal debate. In contrast to the earlier work that expected the affordances of online communication to lead to a transformative, decentralized, and diverse exchange of information online (Bennett 2003), more recently, media diversity has been discussed in the context of algorithmic curation (“filter bubbles”; Pariser 2011) or like-minded friend groups (“echo chambers”; Sunstein 2009). Fears of this sparking a vicious circle leading ultimately to fragmented societies (Levendusky 2013), polarization (Stroud 2008), and the spread of false information (Ciampaglia 2018) have been prevalent in public and academic discourse.

Research on this topic has been held back by the lack of conceptual clarity about media diversity and by a slow adoption of methods to measure and analyze it. Our core argument therefore is that we need to integrate insights from research on diversity in journalism, communication science, law, and computer science to overcome both problems (cf. Möller et al. 2018). For instance, media diversity is seen as crucial element or even synonym of media pluralism and therefore falls under the core principles that are studied by scholars of media policy (Karppinen 2013). Social scientists (including communication and political science) are interested in closely examining media content and what shapes it as well as attitudinal consequences of the diversity of media diets (McQuail 1992). And lastly, in the field of recommender systems design and computational science the incorporation of diversity measurements for the evaluation of news recommender systems has received increasing attention in the last few years (Kunaver and Požrl 2017). There may be other disciplines in which media diversity can be addressed, but usually diversity research in other disciplines either has a different focus and does not focus on media (e.g., sociology); therefore, we limited our scope to the disciplines in which – according to the best of our knowledge – there are substantive enough subfields that deal with media diversity.

Considering that these fields employ not only differing methods but also often operate with vastly different theoretical frameworks and vocabulary, diversity research is becoming increasingly fragmented. To profit from interdisciplinary contributions, though, studies need to employ comparable definitions and measures of diversity.

Therefore, this study aims to (1) give an overview of conceptualizations and operationalizations of media diversity in different fields using a systematic literature review. Especially as diversity can be seen as having different dimensions and levels of analysis, giving an overview and synthesis of the current state of research is of importance to show limitations and “blind spots.” This can serve as input to (2) a united framework that identifies important conceptual and methodological approaches needed to study media diversity in an online environment that is increasingly characterized by personalization and individual supplies of media content. We show important aspects that should be put more into focus when doing truly interdisciplinary research into media diversity that combines theoretical insights with innovative methodological ideas.

A Systematic Overview of Current Theorizing and Research on Media Diversity

To systematically analyze and organize the literature across several fields, we distinguish between the conceptual and the empirical level. On both levels, there is a lack of consensus on what media diversity is or should be. The largest gap, however, is evident when it comes to link conceptual and empirical work, with very few studies spanning across fields.

Conceptual level. It is commonly accepted that media diversity plays an important role for democracy (McQuail 1992). Yet, how the concept should be defined, what it should include, and how it should be measured is far from clear. This fragmentation and ambiguity hinders scientific progress, since shared conceptualizations help to understand conflicting findings and improve the testing of relationships between concepts (Watt and Van den Berg 1995). Diversity can be seen as “mega-concept” (McLeod and Pan 2005) that is very much entangled with several other concepts and is inherently normative, which poses a potential communication problem between different strands of literature. However, to sensibly use media diversity in empirical studies, it must be brought from a high level of abstraction to a more concrete level. One way of doing so could be to specify diversity of media content as “heterogeneity of media content in terms of one or more specified characteristics” (Van Cuilenburg, 1999). More specifically, media diversity can be examined regarding the content (e.g., what is written in newspaper articles), the structure (e.g., the number and distribution of different outlets), and – the newest of perspectives on media diversity – regarding what an individual is exposed to (Napoli 2011). This idea of focusing more on the composition of a media diet of individual news consumers ties in with research on media repertoires (Peters & Schrøder 2018; Hasebrink and Popp 2006).

Media diversity can be seen as inherently normative concept, and the way it is conceptualized and operationalized very much depends on the normative framework applied (Helberger et al., 2018). Diversity is not an end in itself, but can be used as means for reaching democratic goals such as an informed citizenry or an inclusive public discourse. Depending on the specified goal, the benchmark for assessing whether a “satisfying” amount of diversity is reached and how this can empirically be tested changes (Raeijmaekers & Maeseele 2015). Thus, the conceptual definition of diversity and its normative underpinnings significantly affects how it is operationalized and how findings are interpreted.

Empirical level: three steps in the research process. Once the boundaries of definitions and normative theories are specified, one can proceed to translate abstract concepts into empirical operationalizations. Three steps of the research process are important for the assessment of media diversity (Van Cuilenburg 2000): data gathering, measurement, and analysis.

The first step is the data gathering process – deciding which media (newspapers, social media content) are examined on which level of analysis. The second is measurement – selecting dimensions on which the media content should differ such as actors, viewpoints, topics, or structure (Hoffmann-Riem 1987; Napoli 1999; Rössler 2007) and extracting the relevant information from the media content. Specifically when studying media diversity in the online environment with ever-growing amounts of news, the choice for an appropriate method is far from trivial. Especially the usage of (semi-)automated methods is of interest here: Coding actors in a text can be done via manual annotation or via entity recognition; and similarity between texts can be assessed by a human or by automatically comparing vocabularies. Investigating the status of using (semi-)automated methods for diversity assessment can help to show what gaps still need to be filled to scale up research into media diversity.

Lastly, in the data analysis step it needs to be decided how to use the gathered and annotated data to assess the degree to which the media content is diverse and whether the observed variance is sufficient. This last step is impossible to do without linking back to normative goals set in the definitory stage; otherwise no meaningful assessment of diversity can be derived.

A Systematic Review

As we have seen, diversity research seems to suffer from a paradoxical situation: On the one hand, it is one of the most prolific areas of research, attracting attention not only from various subfields within communication science and journalism studies, but also from disciplines reaching from law to computer science. On the other hand, it is exactly this popularity that, in fact, hinders further progress: The differences on both the conceptual and the empirical level have become so great that it has become hard to learn from each other.

In an attempt to bring the wide variety of studies on diversity together, we conducted the – to our knowledge – first systematic review of media diversity to examine the conceptualization, operationalization, and findings on diversity in media (and especially news) content. We performed a database search for publication title, abstract, and keywords in three electronic databases that cover a range of scientific fields (scopus, Web of Science) and one communication specific (Ebsco Communication & Mass Media Complete), further limiting the results to relevant fields (social sciences, law, computational sciences). The search string contained news- and media-related words, including social networks (“news*” or “Facebook” or “Twitter” or “reddit” or “media” or “journalis*”). It did not include “television” or “radio” as the focus of this research is put on text-based content. Additionally, the general term “social network” could not be included since it led to a sharp increase in search results due to being used in other fields of research and contexts. The second group of terms describe diversity in its various forms (“divers*” or “plural*” or “serendip*”). Diversity is the term mostly adopted in communication science, while pluralism is mostly used in legal scholarship and serendipity is mostly used in computational science as a diversity-related term. It additionally included concepts that usually require the measurement of diversity as outcome variable (although not specifically naming it as such) – “filter bubble*” and “echo chamber*.” Wildcards were employed to account for plurals and variations in spelling. The search results were limited to articles published in peer-reviewed journals in the last two decades (1998–2018) to account for the transition from offline to online environments and the use of (semi-)automated methods. The database search was the first of several selection steps in the systematic review process (Figure 1).

Figure 1. Flow diagram of the systematic review process.

After that, topic codes of journals were used to further narrow the selection to relevant fields (social/communication science, law, computational science). Additionally, regular expressions in a Python script were used to filter out incorrect matches due to databases stemming keywords (“news” became “new”). Finally, the abstracts were screened manually according to two inclusion criteria: (1) media (content or market) were the main focus of the analysis and/or the theoretical argument made, and (2) a connection between diversity-related words and media content or market was given. This includes empirical as well as theoretical articles. It does not include fictional formats (games, entertainment movies) or staff diversity (i.e., in newsrooms).

Coding Procedure

The articles were each coded for a number of variables, beginning with whether an empirical study (collection and analysis of data) was conducted or whether a theoretical discussion was made. In a second step, all diversity-related terms appearing in the article were coded – if the term was substantially discussed (i.e., appearing more than once outside of footnotes or references).

The first set of variables aimed at how articles defined and conceptualized diversity: It was coded whether an explicit definition was given, whether (and which) citations were used for it, and its full text was recorded. After this, the level of analysis at which diversity was discussed was coded. The first aspect of the level, whether it was internal or external, ranged from “One media article” to “The media market overall”. Following this, the second aspect (supply vs exposure) was coded, classifying articles as supply oriented, audience-oriented, or both. At the end of this block it was coded which dimensions of diversity were mentioned (here again only if substantively discussed). It could be related to actors, viewpoints, topics, or outlets. Additionally, other dimensions could be filled in and for each selected dimension the specific name given to it in the article was recorded. This allows for investigating whether differences in naming the same dimensions occur. For theoretical articles the coding procedure ended at this point, for empirical articles the measurements and results were further coded in a second and third block.

For every dimension of the diversity-related concept that was measured the general method used to assess it was recorded (e.g., content analysis, survey). Furthermore, it was coded whether (semi-)automated methods were used for assessment and for those several quality-criteria (mentioning metrics like recall and precision) were registered. In a last step, it was assessed whether citations were given for the methods used, whether a mathematical formula was given, and the full methodological procedure was recorded.

Intercoder reliability calculations regarding the inclusion of articles in the sample were made between the first author and a researcher with expertise in the field of media diversity research based on a random sample of 100 publications from the initial search results (a sufficient sample size according to Riffe, Lacy, and Fico (2014, 111), and indicated very good agreement (Cohen’s κ =.83; Landis and Koch 1977). As a result of this initial screening, 298 publications were selected for the second step of the coding process. The full PDFs of the publications were retrieved (for 9 no access was possible) and duplicates as well as non-English publications were excluded (11 in total). After coding whether the criteria of the first stage (media and diversity criterion) were fulfilled when taking the whole text into account, 206 articles were analyzed. This accounts for 2.29% of the originally retrieved (thematically filtered) publications, slightly below medium precision in systematic reviews (Sampson, Tetzlaff, and Urquhart 2011).

Results and Discussion

Definitions and Normative Frameworks

In total, 147 of the coded articles conducted an empirical study, and 59 were of theoretical nature. Looking at the distribution of articles per year already indicates a sharp increase in the interest in diversity especially after 2012 (see Figure 2). The last 5 years account for almost two thirds of all the articles found to fit the criteria since 1999. This increase in publications can especially be explained due to the heightened interest in concepts related to exposure diversity that were coined in the last decade (filter bubbles, echo chambers) – but also for more supply oriented studies a clear increase can be seen. In total, 174 of the 206 studies mentioned diversity and 48 referred to pluralism – some of them including both terms. Filter bubbles (13) and echo chambers (12) were mentioned less, but are also newer terms. Serendipity was only referred to in three of the articles.

Figure 2. Amount of studies mentioning a diversity-related term.

Definitions

In total, 116 of 189 studies that mention diversity or pluralism also give a definition of the concept¹– although clear differences between empirical (54.4% of studies give a definition) and theoretical studies (78.2%) can be identified. Thus, especially in empirical research, it is a common phenomenon to use diversity as term without a clear definition. In those cases, it is used as buzzword term (Raeijmaekers & Maeseele, 2015) without conceptualizing it.

Several scholars specifically address whether pluralism and diversity should be distinct or interchangeable concepts. Some argue that diversity is a measure of media content while pluralism focuses on media structures (Duncan 2015^A) or see diversity as empirical and pluralism as ideological concept (Raeijmaekers & Maeseele 2015^A). However, most studies use the terms interchangeably (see e.g., Ciaglia 2013^A; Hibberd 2007^A; Hoffman et al. 2015^A; Woods 2007^A). Looking at the articles which use one term or the other reveals that the term pluralism is mostly used in articles related to law and political science (e.g., Allen, Connolly, and Heap 2017^A; Czepek and Klinger 2010^A; Gibbons 2015^A) while diversity is more used in communication and computational science. The difference can thus rather be seen as distinction between fields than necessarily a difference in meaning – in the following the two terms are used interchangeably unless explicitly stated.

Standing on the Shoulders of Giants

Looking at the studies that give a definition of media diversity or pluralism reveals that often it is much less a question of how people define diversity but rather who they cite. In most cases, a short overview is given over past conceptualizations of diversity and its multifacetedness is stressed (Helberger 2015^A, Karppinen 2006^A).² However, often the theoretical work ends here – usually no additional contribution is made to the definitions. Thus, one way to understand how diversity is defined is not necessarily to classify the definitions made in the sample but by seeing who is cited and whether the most referenced works do differ in their conceptualizations. Only after this, conceptualizations substantially adding to those widely used definitions and what they might add to the discussion are presented.

Of the 116 studies giving a definition, 87 made a reference to existing definitions. They refer to 184 different sources. Although this shows quite some variance, there are still central studies standing out due to appearing frequently, as displayed in Figure 3: Each vertex (node) of the network represents an article and each arrow (edge) leads from an article to the sources it cited. The size of the nodes represents the number of citations it received (indegree). Only citations specifically made for the definition of diversity or pluralism are considered. It therefore does not – like a general citation network – depict all references made, but only those used for conceptualizing the construct. It shows that eight studies have more than five citations (listed in the legend).

Figure 3. Citation network.

Taking a closer look at those central sources shows that they all made overall conceptualizations of diversity with different dimensions and can be placed in the field of journalism and communication science. This might be due to the fact that in social sciences in general the type of definitions that were coded in this study are most common – but also show a disconnect between the fields since the definitions made in legal and computational scholarship can mostly be found in the disconnected parts of the network, not finding many citations in communication and journalism research. They stress the multifacetedness of the concept and rather define certain elements of it than focus on an overarching definition of diversity or pluralism. The three most cited studies are Voakes et al. (1996), Napoli (1999) and McQuail (1992) – see Table 1 for an overview of the articles citing them, the different dimensions of diversity they addressed, and how those dimensions were defined in the articles.

Table 1. Definitions in central studies.

Study	Cited by*	Dimension	Definition
Voakes et al. (1996)	7, 8, 31, 77, 95, 102, 111, 116, 130, 131, 156	Source	“dispersion of the representation of affiliations and status positions of sources used to create a news product. The more even the dispersion of the representation among source affiliations, status, and proximity in a given story or medium, the greater the diversity” (p. 583)
		Content	“dispersion of representation of ideas, perspectives, attributions, opinions, or frames within a news product, and within the context of one particular issue.” (p. 585)
Napoli (1999)	8, 21, 26, 37, 82, 83, 88, 89, 95, 98, 103, 108, 125, 136, 137, 166, 168, 174, 176, 188	Source	Content Ownership: “producers of the programs that run on the individual cable channels” (p.12) Outlet Ownership: “owners of the individual channels that are carried on the cable systems” (p. 11) Workforce diversity: “diversity of the entire workforce within media outlets” (p. 13)
		Content	Program-Type: “category designations given to radio formats, cable channel formats, and individual television programs” (p. 18) Demographic: “racial, ethnic, and gender diversity of the people featured within electronic media programs” (p. 21) Idea-Viewpoint: “diversity of viewpoints and of social, political, and cultural perspectives” (p. 22)
		Exposure	Horizontal: “distribution of audiences across all available content options” (p. 26) Vertical: “diversity of content consumption within individual audience members” (p. 26)
McQuail (1992)	75, 82, 83, 87, 93, 94, 95, 100, 116, 150, 166, 175, 188, 191, 198, 203	Content	Political: “differing and conflicting interests which seek public expression” (p. 150) Geographical: “serve local needs, as well as to help integration into the wider society” (p. 151) Social-Cultural: “significance of sub-cultural differences based on gender, generation and many kinds of special interest” (p. 152)
		Structure	External: “the full range of relevant differences (…) matched by an equivalent range of separate and autonomous media” (p. 145) Internal: “a wide range of types of content or points of view is offered by the same channel (s)” (p. 146)
		Stages	Source: “range and degree of access and/or exclusion experienced by alternative voices in society” (p. 156) Channel: “number of channels and the degree to which they are different from each other” (p. 156) Content as sent: “universe of content offered by a set of channels or a single channel (over time)” (p. 157) Content as received: “‘content as sent’, weighted according to the size of audience reached” (p. 157)
		Perspective	Reflection: “reflect the prevailing differences of culture, opinion and social conditions of the population as a whole” (p. 144) Access: “separate ‘voices’, groups and interests which make up the society can speak to the wider society” (p. 144) Choice: “a variety of range of products or services available to them [the consumers]” (p. 145)
		Performance	Equality: “all relevant candidate groups or interests have an equal share of access to media channels” (p. 147) Proportionality: “the ‘media distribution’ (e.g., of content voice and image) should match the ‘social reality’ distribution” (p. 148)

* Numbers refer to Appendix A, each reference examined for the literature review got assigned a number for better readability in the table.

Voakes et al. (1996) only concentrate on one media outlet and specifically focus on the distinction of source (who is mentioned) and content (what perspective is taken) and do not address anything but the media content. Indeed, some of the articles citing this are specifically aiming at making a clear distinction between actor and viewpoint diversity. Especially Masini and Van Aelst (2017) and Baden and Springer (2017) expand on this in a very detailed way, further showing why it is important to keep those two dimensions separately.

Napoli (1999) subsumes what Voakes et al. (1996) described under “Content diversity” and adds a topic-element to it. However, he additionally mentions the structural component – terming it source diversity – which includes everything concerned with how the content was made and by whom. Lastly, he adds exposure diversity as a third component. Exposure diversity is also further discussed and elaborated on by other scholars, including Helberger et al., Napoli (2011), and van der Wurff (2011).

By far the most detailed account of diversity is given by McQuail (1992) who talks about content and structural dimensions and also mentions exposure diversity, naming it “diversity as received”. Additionally, he adds elements pointing towards the different goals that can be addressed with diversity: It can be seen from a perspective aiming at society (reflection), actors that want to be heard (access), and the audience (choice). He further already gives standards of performance (equality or proportionality) that diversity can be evaluated against. Some later theoretical work takes similar approaches, further elaborating on goals and performance measures (see e.g., Van Cuilenburg 1999; Karppinen 2013; Helberger et al. 2018).

One additional form of defining diversity can be found in the studies that usually do not cite any other scholars for their definition or that only cite methodological studies (mostly those shown in Figure 2 that are not connected to the rest of the network). They use a more procedural approach – defining a concept by the procedure of the measurement. This can be done by giving a formula or by statements like: “We compute the political diversity of news articles […], which is defined as the (political) entropy of the news articles associated with the user” (An et al. 2014, 9). These kind of definitions are rather common in more computationally oriented research.

Thus, what can be seen are different strategies for addressing and defining media diversity at different levels of theoretical abstraction: Purely empirical approaches who only use a procedural definition, seeing diversity assessment as measurement problem. On the next step, some short definitions of concrete content features are given – as demonstrated by Voakes et al. (1996), who are almost exclusively cited in empirical studies. On the next level not only the media content is considered, but also the way the content is produced (e.g., ownership) and the way it is received (exposure) – as Napoli (1999) demonstrates. Lastly, the different goals (from various perspectives) and benchmarks of assessment of diversity can be of interest as reflected by the work of McQuail (1992).

New ideas: Apart from these very broad classifications and strategies, some definitions specifically contribute important new elements: The audience can now also be seen as co-creator of news, changing the traditional notion of media systems as a supplier of diverse messages, broadening the realm of content that can be classified as media content (Cortesi and Gasser 2015^A). New genres such as citizen journalism or news blogs especially influence the structural aspects of media production. The question of “what is news” and what to include in analyses especially at a higher level of analysis (i.e., media market) becomes more difficult to answer.

Diversity has also been addressed in terms of content richness – whether elements beyond texts (videos, photos, audio, hyperlinks) are used in online media (Carpenter 2010^A; Yang et al. 2017^A). The online news experience becomes increasingly multi-modal, requiring to research diversity beyond text and also for example classifying the content of photos accompanying a news text.

Lastly, it is important to also take the “diversity experience” (Hoffman et al. 2015, 1363) or perceived diversity into account – meaning that environmental factors (supply of content) and behavioral factors (exposure to content) are complemented by how the individual processes and perceives the received information. Hoffman et al. (2015) use social cognitive theory (SCT) to stress that motivation, awareness, and ability of the receiver of information are essential for understanding how diversity is perceived. According to them, the main challenges for diversity in an online environment are not necessarily related to the supply or exposure anymore (as there are endless possibilities for content online) but rather related to what the user is actually willing and capable of experiencing. Especially in terms of taking the abundance of information into account, this more psychological take on media diversity is important to consider.

To summarize, we only identified limited theoretical work regarding the definition of diversity or pluralism. When defining diversity, most articles either give a short overview of past work without adding further to those conceptualizations or use a procedural approach. Thus, the main strategies for defining diversity were deducted from central studies, showing different levels of theoretical abstraction that can be employed. Only the last level (as illustrated by McQuail 1992) also reflects on different perspectives and measures of performance, while the others are mostly concerned with empirical measurements.

Lastly, approaches adding new elements to those established definitions were introduced, stressing that online media (1) blur what is classified as media content due to possibilities of co-creation by users (2) increasingly require research of different media modes and (3) emphasize the need for taking the cognitive level of perceived diversity into account. However, only few studies address those new challenges for media diversity research – the main development of theories took place in the 1990s, since then new developments have only been sparsely addressed and incorporated in theoretical frameworks.

Normative Frameworks

The societal importance of media diversity greatly depends on the function it is assumed to have for democracy - something that has been discussed in detail in the legal and policy field. However, it remains a question in how far the connection to other domains has also been made, bridging the gap between theoretical and empirical research. Overall, it is very common to at least briefly refer to democracy or normative ideals. 42% of the empirical and 74% of the theoretical articles mention diversity or news/media in the context of democracy or normative ideas.³ However, most articles only limit this to one generic sentence. It is common to state that diversity is important for democracy to stress the societal relevance of research in introduction or discussion – however, without further specifications as to the type of democracy that is used. This is only done by few scholars (e.g., Humprecht and Esser 2018^A) who all stress that it indeed matters which democratic framework is applied for the definition, goals, and benchmarks of diversity.

Four articles specifically lay out the different normative frameworks that can be applied in the context of diversity (Bozdag et al. 2014^A; Helberger 2015^A; Möller et al. 2018^A; Raeijmaekers & Maeseele 2015^A). They all mention four models: the liberal-aggregative and liberal-individual framework as well as deliberative and adversarial/critical models.

Taking a closer look at whether and which of these normative ideas are mentioned in the definitions of diversity in the sample shows that the overall consensus is to refer to a liberal-aggregative conceptualization of diversity: Those articles often mention the “marketplace of ideas” (e.g., Haim, Graefe, and Brosius 2018^A; Jacobi, Kleinen-von Königslöw, and Ruigrok 2016^A; Masini and Van Aelst 2017^A; Raeijmaekers & Maeseele, 2015^A; Shumow and Vigon 2016^A; Vos and Wolfgang 2018^A), the importance of a variety of sources and viewpoints and the function of media to reflect the diversity of society. This preference for the liberal conceptualization can be explained by the high amount of studies from the U.S. in the sample, where the liberal, market-oriented perspective is most widely accepted – and because it is the most popular among highly cited scholars (Napoli, Voakes).

The second model, deliberative democracy, is specifically referenced in some articles (e.g., Bozdag et al. 2014^A; Haim, Arendt, and Scherr 2017^A; Lörcher and Taddicken 2017^A; Rohlinger and Proffitt 2017^A; Weeks, Ksiazek, and Holbert 2016^A), but in general the idea of the public sphere and inclusive, rational debates finds only little explicit resonance. The other two perspectives (adversarial and liberal-individual) are almost not present in our sample: Some specifically stress that minorities should be more strongly represented (Baum 2013^A; Binderkrantz, Bonafont, and Halpin 2017^A; Dwyer and Martin 2017^A; Shumow and Vigon 2016^A) and examine whether elite actors are dominating the discourse – however, this can only be seen as rather weak link to the adversarial viewpoint in the sense that it aims at promoting less heard voices. The value of critical and conflict-laden discussions is only sparsely mentioned (Möller, et al. 2018^A), showing that the antagonistic democratic view – although theoretically valid – is not adopted in many studies.

The perspective of individual autonomy was applied in studies that stress that the individual users’ choice is of higher importance in the online environment (Hoffman et al. 2015^A; Horowitz and Nieminen 2017^A). It could further be reasoned that articles focusing on perceived diversity take this standpoint as they are moving the user in the center of analysis (Van Den Bulck et al. 2016^A; van der Wurff 2011^A; Yang et al. 2017^A). One study shows that journalists often take this perspective, indicating that they have to primarily serve the needs of individuals with diversity as a welcome by-product (Vos and Wolfgang 2018^A).

Overall, many studies do not or only superficially mention normative frameworks, and those that do primarily refer to liberal-aggregative ideas of democracy. However, the references to that framework mostly remain limited to common metaphors. Deliberative, individual autonomy and antagonistic frameworks are proposed in classifications of democratic frameworks but do not find many takers in research practice. This might be related to difficulties in finding concrete benchmarks for assessing diversity in those frameworks.

Benchmarks

Benchmarks are essential for evaluating media diversity: An answer to the normative question “Do we have a good amount of diversity?” is far from trivial and needs considerations going beyond mathematical aspects. A rather naïve way of thinking of evaluating diversity is the “the more the better” assumption – which is indeed explicitly mentioned in some articles (Carpenter 2010^A; Vos and Wolfgang 2018^A) and far more often seems to be implied when conclusions state sentences similar to “Newspaper X performs better regarding diversity than newspaper Y because they mention more different parties”. Those articles predominantly appear in communication or computational science, showing that expertise gathered in the field of policy has often not been incorporated in other disciplines. However, Van Cuilenburg (1999) stresses that “media diversity should always be compared with relevant variations in society and social reality” (190) – benchmarking means anchoring the diversity in the media to the diversity of society. The main benchmarks needed for each framework are shown in Table 2. They range from mirroring society (liberal-aggregative), consumer satisfaction (liberal-individual), access to a rational debate with equal shares (deliberative) to promoting minority voices and emotional tone (adversarial/critical).

Table 2. Normative frameworks and benchmarks.

Normative framework	Focus	Main goals	Benchmark
Liberal aggregative	Market	Reflection of social heterogeneity	Mirroring society (adhering to market shares, polls, voting results) Reflective diversity and avoidance of bias
Liberal-individual	Consumer	Autonomy of the individual	Consumer satisfaction No restriction to individuals’ autonomy to choose
Deliberative	Public Sphere	Inclusive public debate Civilized tone Discover all possible perspectives	Open diversity (equal shares) of viewpoints and entities Inclusion of all actors in the debate (access) Rational arguments in favor of emotions
Adversarial	Alternative voices	Promoting minorities Challenging the existing system Dissent, conflict, contestation	Share of minority voices and small groups in the news Amount of attention dedicated to non-governmental sources Emotional tone in favor of rational argument

Table derived from several studies in the sample (Bozdag and van den Hoven 2015^A; Helberger 2015^A; Möller et al. 2018^A; Raeijmaekers & Maeseele 2015^A).

Empirical Operationalization and Assessment

The operationalization of diversity mostly focusses on journalism and communication science as well as computational science – since in legal scholarship the empirical assessment plays only a subordinate role and not many empirical articles were found in this domain.

Data Gathering

The first step of the data gathering process is to decide which media to look at when assessing diversity. More than half (55%) of the studies still focus on traditional mass media such as offline newspapers, television, and radio, or a mix of different media in the market. However, especially during the last decade, newspapers are examined in different versions (online and offline) and also other forms have been studied: online only news websites and news aggregators/recommender systems (Gu, Dong, and Chen 2016^A; Haim, Graefe, and Brosius 2018^A), consumer-produced content (e.g., social media; Beam, Hutchens, and Hmielowski 2018^A; Bechmann and Nielbo 2018^A), and alternative sources of information (e.g., blogs; González-Bailón and Paltoglou 2015^A; Park et al. 2013^A).

The second step is to select a level of analysis. The classical distinction to be made is between internal and external diversity (McQuail 1992). While the former describes the diversity of content within one media outlet, the latter refers to the amount of sources available in the media system or sectors of the media system and how distinct they are. In past media diversity research, internal diversity played a rather minor role, most studies focus on external diversity by comparing or examining multiple outlets in one sector (i.e., different newspapers, see Appendix Table B2), addressing concerns that an unbalanced market structure is the biggest threat to a pluralistic media diet.

However, what becomes apparent is that some diversity-related terms are mostly used in the context of internal diversity: Filter bubbles and echo chambers are almost exclusively researched on single platforms (e.g., Facebook; Flaxman, Goel, and Rao 2016^A) or outlets. This poses several challenges: When the effects of like-minded friendship groups or algorithmic personalization are always only examined within one outlet, only statements about that single part of a persons’ media diet can be made. However, it is highly unlikely that all news intake is provided by one single outlet. Thus, the question whether someone is in a filter bubble or echo chamber requires higher levels of aggregation. In contrast, media markets or media in general are often discussed in theoretical articles from political science and law (talking about pluralism) or more economy-centered publications about ownership and concentration.

Some of the articles could not be placed in the traditional distinction of internal diversity and external diversity: They look at the whole online diet of individuals by inspecting weblog data (Dvir-Gvirsman, Tsfati, & Menchen-Trevino, 2016^A) and other digital traces (Flaxman, Goel, and Rao 2016^A) or examine individual news recommendations from websites and aggregators (Trampuš et al. 2015^A; Haim, Arendt, and Scherr 2017^A; Kiritoshi and Ma 2016^A). This shows that media consumption becomes more and more personalized, highlighting the importance of the distinction between supply and exposure diversity. It has become more of a concern in the last few years (e.g., Helberger et al. 2018; Kim & Kwak 2017; Hoffman et al. 2015^A) and addresses that it is of importance not only how diverse the supply of diverse content is but also, whether individuals are actually exposed to this diversity (Napoli 2011). While the former is supply oriented (what is provided by media outlets), the latter is inherently audience-oriented (what is consumed by the audience). Supply diversity can simply be approached through analyzing the media content, but exposure diversity requires additionally a measure of which of the content a person actually was exposed to or consumed.

This increasing shift towards an audience-centered perspective can also be seen when looking at whether the exposure or supply perspective is taken. Overall, the supply perspective is clearly dominating, with 142 of 206 articles taking this media-centered take on diversity. 40 studies are solely focused on exposure, and 16 combine both. Three studies used the concept of “perceived diversity,” having participants assess the diversity instead of measuring it in the content. Supply oriented perspectives dominate overall, but looking at the distribution over time (see Figure 1) shows a clear trend: While it was only addressed in two studies before 2011, it has been mentioned 28 times after. The discussions about algorithmic curation influencing the media environment and thereby diversity, sparked by Sunstein (2009) and Pariser (2011) can be seen as one of the influencing factors of making this form of diversity more popular.

Measurement

We also identified a wide range of approaches to translate specific sub-dimensions of diversity operationalized into measurable content features across sub-fields and disciplines. There are several levels of specificity in the various studies. Generally speaking, the terms “content diversity” or “information diversity” should not be conceptualized as dimensions since they do not offer a precise enough definition of what is included and meant by them (e.g., it has been used to talk about topics (Kordus 2014^A), viewpoints (Sup Park 2014^A) and entities (Lozano 2006^A)). Rather, content diversity can be described as encompassing the elements entities, topics, and viewpoints and as being opposed to structural diversity – similar to the original conceptualization by Napoli (1999). An overview of the sub-dimensions in all analyzed studies is shown in the Appendix Table B1.

Table B1. Level of analysis and dimensions (N = 206).

Level of analysis	Entity	Topic	Viewpoint	Structural	Other
One outlet	Sources: Kiernan (2016); Rodgers, Thorson, and Antecol (2000) Voice: Lee, Sang, and Xu (2015) Actor: Merry (2016)† Company: Jonkman et al. (2018) Groups: Binderkrantz (2012) Minority: Bozdag et al. (2014)	Topic: Möller et al. (2018); Zafar et al. (2015); Bechmann and Nielbo (2018)†; Li and Thorson (2015) Issue: Tan and Weaver (2013) Information: Kang and Lerman (2017)	Viewpoint: Carlson (2007) Perspective: Jacobson, Myung, and Johnson (2016); Ross and Bantimaroudis (2006)† Opinion: Lorentzen Gunnarsson (2015)† Political: Merry (2016)†; An et al. (2014)† Attitudes: Grömping (2014)† Partisan: Colleoni, Rozza, and Arvidsson (2014)† Frame: Dahinden (2002) Polarization: Beam, Hutchens, and Hmielowski (2018)† Stance: Lee, Sang, and Xu (2015)	Source: Bozdag et al. (2014) Media: Lee (2002)*	Message: Luo, Pan, and Zhu (2017) Output: Bozdag et al. (2014) Information Sources: Himelboim (2014) Linguistic: Perlman and Amaya (2013) Tone: Möller et al. (2018)
Multiple outlets (one sector)	Sources: Binderkrantz, Bonafont, and Halpin (2017); Yang et al. (2017); Zeldes, Fico, and Diddi (2012); Oh et al. (2012); de Swert and Wouters (2011); Correa and Harp (2011); Scott, Chanslor, and Dixon (2010); Day and Golan (2005); Lacy et al. (2013, 2012); Matthews (2013); Armstrong (2006); Kim and Jahng (2016) Voice: Young and Dugas (2012) Actor: Humprecht and Esser (2018); Udris, Eisenegger, and Schneider (2016) Gender: de Swert and Hooghe (2010); Freedman, Fico, and Love (2007) Perspectives: Yoon (2013)*	Topic: Yang et al. (2017); Trilling and Schoenbach (2015); Shumow and Vigon (2016); Rodgers et al. (2004) Issue: Udris, Eisenegger, and Schneider (2016) Genres: Fernández-Quijada (2017) Content: Kordus (2014) Coverage: Young and Dugas (2012) Program Vergeer, Eisinga, and Franses (2012); Wonneberger, Schoenbach, and van Meurs (2013) Themes: Correa and Harp (2011) Agenda: Peter and de Vreese (2003); Culbertson (2007); Frames: Baum (2013)	Viewpoint: Humprecht and Esser (2018); Trilling and Schoenbach (2015); Lõhmus et al. (2013); Weeks, Ksiazek, and Holbert (2016)† Perspective: Rohlinger and Proffitt (2017) Opinion: Park, Wang, and Pinto (2016); Ashley (2015); Duckett and Langer (2013) Idea: Rohlinger, Pederson, and Valle (2015); Kordus (2014) Connotation: Woods (2007)* Political: Kim and Kwak (2017); Rodgers et al. (2004); Fletcher and Nielsen (2018)† Slant: Flaxman, Goel, and Rao (2016)† Content: Sup Park (2014); Day and Golan (2005) Frame: Baden and Springer (2014); Huang (2010); Park et al. (2013); Entman (2006); Roggeband and Vliegenthart (2007) Standpoint: Schafraad, Wester, and Scheepers (2013) Ideology: Pineda and Almiron (2013)	Source: Napoli (2002) Channel: Flaxman, Goel, and Rao (2016)†; Sup Park (2014); de Swert and Wouters (2011); Yuan (2008)	Churnalism: Saridou, Spyridou, and Veglis (2017)* Media types: Yang et al. (2017) Duplication: Shumow and Vigon (2016) Country: Hanusch (2014) Content: Napoli (2002) Geographical: Rodgers et al. (2004); Humprecht and Esser (2018)
Multiple outlets (more sectors)	Sources: Van Leuven, Heinrich, and Deprez (2015); Bagashka (2014); Tiffen et al. (2014) Entity: Augenstein, Derczynski, and Bontcheva (2017) Parties: Jacobi, Kleinen-von Königslöw, and Ruigrok (2016); van Hoof et al. (2014) Polyvocaliy: Thorsen and Jackson (2018)	Topic: Lörcher and Taddicken (2017); González-Bailón and Paltoglou (2015); Carpenter (2010); Bae (2000); Smyrnaios, Marty, and Rebillard (2010)* Issue: van Hoof et al. (2014) Themes: Seo (2018)	Idea: Czepek and Klinger (2010)* Opinion: Zhao (2016); Dubois and Blank (2018)† Ideology: Benson, Neff, and Hessérus (2018) Issue Agenda: Lee (2007a) Interpretation: Lörcher and Taddicken (2017)	Source: Nechushtai and Lewis (2019)† Media: Dubois and Blank (2018)† Outlet: Czepek and Klinger (2010)* Ownership: Benson, Neff, and Hessérus (2018); Trappel (2008); Dwyer and Martin (2017)	No Dimension: Welbers et al. (2018); Herrmann (2016)*; Haim, Graefe, and Brosius (2018)† Token: Augenstein, Derczynski, and Bontcheva (2017) Media Type: Carpenter (2010); Trappel (2008) Lexical: Smyrnaios, Marty, and Rebillard (2010)* Cultural: Dwyer and Martin (2017)

Concept described:

^* Pluralism

^† Echo chamber or Filter bubble

Level of analysis and dimensions (N = 206)

Sub-Dimensions of Diversity

Diversity of entities is a term often employed in computational science. It encompasses people, groups, and organizations. By specifically using a rather technical term it is avoided that this dimension is confused with the viewpoint-oriented dimension, which is the case by studies using the term “voice” (Hibberd 2007^A; Lee, Sang, and Xu 2015^A). It also bypasses another common issue: “Source diversity” is the most-used label for entities, coming from a journalistic perspective of sources in articles and often rooting back to Voakes et al. (1996). Sources are, however – in the tradition of Napoli (1999) – also used to describe different outlets in the marketplace.

A second sub-dimension we identified is topic diversity – this is also the most-used term in the analyzed papers for this dimension and can be seen as a synonym for issue. It avoids confusion with perspectives (e.g., since one study called a topic-related dimension frames; Baum 2013^A) or staying to closely within the terminology of one media type (program (Vergeer, Eisinga, and Franses 2012) or genre (Fernández-Quijada 2017) for television).

The third content-related sub-dimension is viewpoint diversity as a rather neutral term for describing that one topic can be viewed from different angles. It thus avoids value-laden terms such as frames (Dahinden 2002^A; Entman 2006^A) or polarization (Beam, Hutchens, and Hmielowski 2018^A) and is not necessarily limited to the political realm (such as ideology; Benson, Neff, and Hessérus 2018^A).

Lastly, the term structural diversity is used to describe the sub-dimension not related to the media content but to questions about who supplies the information. By that it avoids being reduced to either “outlets” and “channels” or questions of ownership, but rather expresses that it is about structural features of the market proposed to influence diversity.

The most frequently investigated sub-dimension is viewpoint diversity. This is hardly surprising given the often normative connotation of diversity in relation to political views, polarization, or fragmentation. It is the only dimension that is consistently evaluated across all levels from single outlets to market level – as opposed to entities and topics that are only used when analyzing one or multiple outlets. However, it could be argued that specifically on the market level, entities can give insights into pattern of access to the news and public discourse at large. Especially when taking the deliberative or adversarial perspective, the question of who gets to speak to what extent is crucial. In the same realm, specific groups are linked to topics (in terms of issue ownership, Petrocik 1996) – thus the question of who sets the topic agenda and determines what is discussed in the news, puts some entities more on the spot than others.

Several articles did not fit into the previously defined dimensions: They add dimensions that could be of special interest regarding the changes from offline to online content: Medium diversity (i.e., audio, video links) and perceived diversity were already mentioned in the definitions above. Additionally, linguistic and lexical diversity can be analyzed with automated methods judging the syntactic and semantic differences between news outlets. Especially for the assessment of factors such as tone or sentiment such approaches are valuable. Articles naming those diversity dimensions were published within the last 10 years and almost all research online content.

Approaches to Assess Sub-Dimensions of Diversity

Across all studies, most employed manual content analysis to assess sub-dimensions of diversity. We identified only 26 studies that used (semi-)automated methods to extract relevant measurement indicators, 89% of those were published after 2013. Since then, we identified four to five studies each year that relied on automated methods. This indicates that automated methods do not have a prominent space in media diversity research in the social sciences yet, even though they have the potential to advance the scale of research by automatically measuring different indicators. 10 of the 26 articles were published in journals that are specific to computational science and linguistics, which decreases the visibility of those methods to communication scholars.

An important characteristic of the computational studies is that half of them did not mention any attempts to validate that the computational methods actually capture the intended variables, this is particularly striking in those studies using named entity recognition or unsupervised methods such as LDA topic modeling.

Data Analysis

Turning to the assessment of diversity, the frameworks built by Stirling (2007) as well as McDonald and Dimmick (2003) are used, coining the terms “variety,” “balance,” and “disparity” to describe different approaches. The first one, variety, is “the answer to the question: how many types of thing do we have?” (Stirling 2007, 709) and has been termed “single-concept diversity” (McDonald and Dimmick 2003) as it relies on only one indicator (e.g., the number of entities in an article). An overview of the studies using the different approaches is given in Appendix Table B2.

Table B2. Methods and measures empirical studies (N = 148).

Method	Variety	Balance	Disparity	Other
Content Analysis (manual)	Baden and Springer (2014); Benson (2009); Benson, Neff, and Hessérus (2018); Binderkrantz (2012); Lanosga and Martin (2018); Seo (2018); Rohlinger and Proffitt (2017); Masini et al. (2018); Park, Wang, and Pinto (2016); Kim and Jahng (2016); Trilling and Schoenbach (2015); Van Leuven, Heinrich, and Deprez (2015); Lee (2002); Rohlinger, Pederson, and Valle (2015); Bagashka (2014); Dvir-Gvirsman, Tsfati, and Menchen-Trevino (2016); Kordus (2014); Young and Dugas (2012); Zeldes, Fico, and Diddi (2012); Oh et al. (2012); de Swert and Hooghe (2010); de Swert and Wouters (2011); Correa and Harp (2011); Scott, Chanslor, and Dixon (2010); Trappel (2008); Freedman, Fico, and Love (2007); Lee (2007a); Day and Golan (2005); Dahinden (2002); Kurpius (2002); Park et al. (2013); Pineda and Almiron (2013); Matthews (2013); Duckett and Langer (2013); Lacy et al. (2013); Yoon (2013); Schafraad, Wester, and Scheepers (2013); Udris, Eisenegger, and and Schneider (2016); Rodgers et al. (2004, Rodgers, Thorson, and Antecol 2000); Armstrong (2006); Lacy et al. (2012); Thorsen and Jackson (2018); Saridou, Spyridou, and Veglis (2017); Sjøvaag (2014); Wonneberger, Schoenbach, and van Meurs (2013); Merry (2016); Lõhmus et al. (2013)	Binderkrantz, Bonafont, and Halpin (2017)*; Salgado and Nienstedt (2016)†; Gronemeyer and Porath (2014)*; Carpenter (2010)†; Lee (2007b)†; Powers and Benson (2014)‡; Masini and Van Aelst (2017); Humprecht and Esser (2018); Hanusch (2014); van Hoof et al. (2014); Takens et al. (2010); Huang (2010); Woods (2007); Peter and de Vreese (2003); Bae (2000); Culbertson (2007); Humanes et al. (2013); Kiernan (2016)
Content Analysis (automated)	Lörcher and Taddicken (2017); Shumow and Vigon (2016); Smyrnaios, Marty, and Rebillard (2010); Baum (2013); Li et al. (2014); Roggeband and Vliegenthart (2007)	Jonkman et al. (2018)*; Jacobi, Kleinen-von Königslöw, and Ruigrok (2016); An et al. (2014)*	Zafar et al. (2015); Toraman and Can (2015); Trampuš et al. (2015)
Survey	Park (2018); Dubois and Blank (2018); Kim and Kwak (2017); Beam, Hutchens, and Hmielowski (2018)	Moehler and Allen (2016); Dvir-Gvirsman (2015); Fletcher and Nielsen (2018); Wojcieszak and Rojas (2011); Weeks, Ksiazek, and Holbert (2016); Wang, Guo, and Shen (2011)	Similarity: Welbers et al. (2018); Li and Thorson (2015) LDA: Bechmann and Nielbo (2018); González-Bailón and Paltoglou (2015) Entities: Kiritoshi and Ma (2016); Augenstein, Derczynski, and Bontcheva (2017) Linguistic: Maekawa et al. (2014); Smyrnaios, Marty, and Rebillard (2010)	Perceived Diversity: Yang et al. (2017); Zhao (2016); Van Den Bulck et al. (2016); van der Wurff (2011)
Linguistic analysis		Network: Bozdag et al. (2014)* Market: Artero and Sánchez-Tabernero (2015)‡; Ciaglia (2013)‡; Allen, Connolly, and Heap (2017)‡ Secondary data: Vergeer, Eisinga, and Franses (2012); Entman (2006)‡; Tan and Weaver (2013)† Simulation: Möller et al. (2018)*; Bozdag and van den Hoven (2015)	Network: Kang and Lerman (2017); Jacobson, Myung, and Johnson (2016); Colleoni, Rozza, and Arvidsson (2014) Secondary data: Mangani and Tarrini (2018) Search result analysis: Yom-Tov, Dumais, and Guo (2014) Simulation: Haim, Graefe, and Brosius (2018)	Topic probabilities: Luo, Pan, and Zhu (2017)
Qualitative	Qualitative CA: Ashley (2015); Heinrich and Pleines (2018); Herrmann (2016)			Interview: Davies (2018); Vos and Wolfgang (2018); Cornia (2016); Duncan (2015) Qualitative CA: Lorentzen Gunnarsson (2015); Grömping (2014) Observation: Makri et al. (2015) Discourse Analysis: Sup Park (2014)
Other	Market: Engesser and Franzetti (2011) Experiment: Nechushtai and Lewis (2019) Secondary Data: Fernández-Quijada (2017); Yuan (2008) Simulation: Haim, Arendt, and Scherr (2017)			Trace data: Flaxman, Goel, and Rao (2016) Experiment: Urban and Schweiger (2014)

Mathematical formula used:

^* Shannon’s entropy;

^† Simpson’s D;

^‡ Herfindahl-Hirschman Index (HHI).

Measuring diversity in the sense of variety can be seen as the consensus in the field so far. Especially (manual) content analysis is used for gathering indicators whose frequencies and proportions are subsequently reported. This comes as no surprise since especially the method of quantitative content analysis is useful for assessing frequencies. The variety approach is also occasionally used for qualitative content analysis, surveys (making additive indices) and other, less frequently appearing methods.

This clear dominance of measures of variety can be seen as rather problematic (McDonald and Dimmick 2003). When only having the amount elements in different categories, their distributions can be entirely different. Only looking at the distributions without considering the number of elements in the distribution also does not fully capture the different parts of diversity. Lastly, variety does not in itself promote a specific benchmark apart from “the more the better.” As discussed above, this lack of an appropriate benchmark gives only little information about diversity in a normative sense. It can, however, be used as a basis for comparison to distributions of external measures. Thus, to some extent variety can be used for purposes of the liberal-aggregative framework – but should involve interpretation regarding whether the found distribution reflects in some form the societal heterogeneity. Thus far, this is usually not discussed in research.

The second category, balance, looks at distributions of categories. It is aimed at a “dual-concept diversity” (McDonald and Dimmick 2003, 63) since both the number of distinct categories and the evenness of the distribution of these categories are considered. In a first step it is also necessary to take into account the variety of distinct categories, but additionally, some distribution measure needs to be added. This is most often done by using various measures of entropy – the most popular being Shannon’s entropy, Simpson’s D, or the equivalent Herfindahl-Hirschman Index. This strategy takes as benchmark the notion of open diversity – since it is always aimed at flattening the distribution, making the different elements equal. It can thus best be used in the normative framework of deliberative democracy. As Van Cuilenburg (1999) mentions, in rare occasions this can also coincide with reflective diversity – when only two categories are given such as male-female and those are expected to be equally represented in society.

The third way of measuring diversity is by looking at disparity (Stirling 2007). It considers how far from each other two elements are. This is implicitly included when making categories for a content analysis codebook since it needs to be judged how different the categories need to be to be coded as distinct. However, this only can remain a very rough understanding of disparity of elements since it often does not express whether for example two viewpoints are close together or far apart. There are also a few studies in the sample that measure disparity or distance between elements more explicitly – and most use automated content analysis or (computational) linguistic. Measures such as cosine or Jaccard similarity between documents have been used to define how similar or dissimilar the analyzed content is. Two publications look at the repurposing of content by comparing similarities between the articles from news agencies and newspapers or between outlets from one media company (Sjøvaag 2014^A; Welbers et al. 2018^A). In that they contribute to show that the same content can be used in multiple publications due to the common sourcing practices of news outlets. Thus, the sheer amount of content or the distribution of different outlets does not give an indication of how distinct the content is.

The added value of disparity is that it can express whether a person is reading content slightly leaning towards one side of the political spectrum or rather content from the extreme fringes of the spectrum. Especially regarding viewpoints the question of redundancy of standpoints is essential (Baden and Springer 2017). In that it lends itself quite well to the adversarial framework, since it can be used to see how much contestation and dissent can be found.

Some approaches could not be classified as measuring diversity in terms of variety, balance, or disparity: Especially survey approaches and qualitative approaches were concerned with measuring perceived diversity; additionally, participants in an experiment are asked to judge content diversity (Urban and Schweiger 2014^A). Those studies all stress that diversity can also be seen as something that cannot necessarily be only measured by looking at “what is in the content” but that it very much depends on the perspective and perception in how far media are judged as diverse. In that it fits perfectly with the understanding of the individual- autonomy perspective that aims at having user satisfaction as a benchmark of diversity.

In summary, the empirical assessment of diversity shows that in terms of analysis, most studies rely on a single-concept understanding of diversity (variety). When measuring balance (mostly entropy measurements) especially the notion of open diversity can more accurately be expressed and more information can be shown. Apart from these often-used approaches, the notion of disparity should become more prominent, looking at the distance of different elements. It offers opportunities for a more detailed description of differences between the content dimensions measured or structural properties.

Moving Forward: Interdisciplinary Media Diversity Research

This systematic literature review of the past two decades of media diversity research has clearly demonstrated that the concept of diversity is used quite diversely itself. So far, there is little to no overlap between concepts and operationalizations used in the different fields interested in media diversity: the conceptual work and normative considerations from law and policy research are often not incorporated in empirical studies in the field of media use and effects studies, both in communication science and computer science, leading to “the more the better” assumptions without reflecting on the democratic importance of diversity. Likewise, the adoption of computational methods to facilitate data gathering, measurement, and analysis in communication science is rather the exception than the rule. What is more, when computational methods are used, they are often only poorly validated, not taking into account normative benchmarks and considerations.

The fragmented approaches and conceptualizations in the literature illustrate a clear need for consolidation: How should we go forward? First, we need to shift our attention towards a meaningful conceptualization in all sub-fields that takes into account changes such as more and more individualized and personalized media diets and more blurred distinctions between journalistic and interpersonal/citizen-produced content. We therefore suggest to move away from sub-field specific definitions of diversity and focus on an integrated framework that centers around two key terms: the locus and the dimensions of diversity. Our core argument is that on both aspects, the move to digital journalism has created an urgent need for both reconceptualization and new empirical investigation which will require interdisciplinary collaboration to make progress.

First, the locus of diversity refers to which place in the journalistic information chain we are looking for diversity. Traditionally, a distinction is made between Supply diversity and Exposure Diversity. In this distinction, supply diversity studies the total diversity present in the media system and encompasses the structural properties of media systems, media ownership, concentration, and market shares. For questions on media effects, the audience-centered notion of exposure diversity is key (as also stressed by Napoli (2011) and Helberger et al. (2018)). This means we need to conceptualize and measure the diversity of the content used by individual users.

In the era of digital journalism, however, this distinction is becoming more blurred. With news algorithms and other filtering systems offering personalized content, there is no longer a single homogenous news supply for all users. This has made studying media markets increasingly challenging – the standard measures of audience reach and market power used for all offline media are nearly impossible to translate into the converging online context. Moreover, difficulties in gathering reliable and valid usage data and organizing and cleaning it complicates analyses of media markets. Similarly, with many news consumers browsing many different news sources in a context of strong competition for attention, there is an important difference between what news one is exposed to (e.g., what actually appears on the screen, personalized exposure) and what news one consumes (what one clicks on and reads and processes, individual media diet).

Here, methods from the field of computational science can help in getting a systematic overview of the content and providers available and storing them for example by periodically scraping news websites and building comprehensive online archives (see e.g., the project INCA, Trilling et al. 2018). Emerging methods from digital tracking (e.g., Menchen-Trevino 2016) are needed to measure both personalized news exposure and actual consumption, and eye tracking or scrolling data could be used as a proxy for processing. To analyze diversity on an aggregate level we need to move beyond measuring audience shares, but develop measures that can illustrate how diverse media diets are on aggregate, combined with meaningful indicators of the fragmentation of the audience landscape.

Second, we need standardized measures of the core sub-dimensions of diversity. These are diversity in entities, topics, and viewpoints. In addition, linguistic diversity (complexity, tone, or sentiment; Sjøvaag 2014^A; Smyrnaios, Marty, and Rebillard 2010^A) as well as medium diversity (photos, hyperlinks; Carpenter 2010^A; Yang et al. 2017^A) can be used to account for an online landscape that is less bound to traditional patterns of journalistic reporting and where multi-modality influences news experiences. Studies should aim at measuring multiple of those different indicators to better understand how they interact. This requires us to pool resources and collaborate across sub-fields and possibly with non-academic partners to develop valid instruments. If this is not possible, the implications of the limited measurement for the explanatory power of a study need to be acknowledged. Specifically, more emphasis needs to be put on analyzing larger parts of (online) media diets. As has been shown, research into diversity affected by algorithmic personalization so far remains limited to single outlets or platforms (Flaxman, Goel, and Rao 2016^A).

To measure diversity of media exposure, the ideal situation of research would require to analyze what one could call the “media market of an individual,” consisting of all the different media that are consumed – combining offline and online media into one comprehensive account of media consumption. This requires massive efforts from both the researcher and participants since many different methodological approaches and measurement instruments would need to be combined, often making this a non-feasible option. Past work on measuring the media repertoires of individuals gives suggestions on how to approach the challenge of fragmented and multi-modal news media consumption. They often require the combination of quantitative and qualitative approaches – such as tracing respondents’ digital footprints and qualitative interviews aimed at sense making (Peters & Schrøder 2018), while others call for the analysis of secondary data sources (Hasebrink and Popp 2006).

Gathering digital trace data in combination with an analysis of content is needed to understand behavioral patterns of news selection since self-report measures do not adequately capture this (Vraga and Tully 2018). The methodological approaches used so far show a clear focus on content analysis – this further needs to be supplemented by (1) automated and linguistic approaches and (2) surveys and qualitative approaches. Automation and linguistics can especially help with capturing disparity and analyzing large volumes of content as is required due to the continuous flow of information produced every second. They do not play a large role in communication research and journalism studies yet, but can be informed by interdisciplinary research – and here especially the question of validating the results properly needs to be kept in mind. Surveys and qualitative approaches can show aspects of diversity that go beyond the content (van der Wurff 2011^A; Zhao 2016^A). Matters of who gets to access what content, what is used, and how it is perceived, play an increasingly important role.

In analysis, we need to go beyond looking at variety and more strongly take matters of balance and disparity into account. As was shown, the majority of communication science studies limit their assessment of diversity to reporting frequencies without taking distributions or distances into account. Especially disparity is so far a very neglected aspect in communication research – here, approaches from computational sciences and linguistics using measures of distance and overlap (Sjøvaag 2014^A; Welbers et al. 2018^A) can give interesting insights and should be used more (see Stirling (2007) for mathematical procedures).

Input from especially legal studies and (political) philosophy will be needed for a better understanding of the desired level of diversity. Only three of the surveyed articles move beyond an implicit more-is-better position and explicitly problematize that a balance has to be found where diversity turns into dissonance and what should be considered as part of public discourse (Humprecht and Esser 2018^A; Karppinen 2006^A; Raeijmaekers and Maeseele 2017^A). This aims at what the range of included and acceptable opinions should be – who is included in the open diversity measurement and who is not. This stresses that maximal diversity is not necessarily optimal diversity. Especially when considering topics that involve science denialism or radical, anti-democratic ideas, the realm of what is part of the public discourse needs to be set. This requires weighting and balancing the different desiderata stemming from different democratic perspectives: Should in a liberal aggregative framework the whole society be mirrored or only “acceptable” opinions? Is it part of the autonomy of the individual to choose content that goes against a constitution? How inclusive should a deliberative public sphere be? And are those minorities that provoke conflict and dissent with the system always the ones that should be brought forward? Deciding what is part of the public discourse and what not influences normative goals and measurements and should thus be made more explicit.

To answer these questions, it is crucial to take individual as well as societal perspectives into account, creating a hybrid of liberal-individual and deliberative considerations. The current reality of how news presented and consumed online puts individual needs into the focus: The citizen becomes a consumer that has to be satisfied. Diverse media content thus needs to be attractive to users or they will choose something else. However, only choosing the liberal-individual framework for studying media diversity ignores the societal and democratic function of journalism (Karppinen 2013). Especially in the context of debates about fragmented media landscapes, the goal of “satisfying the user” needs to be paired with considerations about how to still enable common spaces of public discussion. One example of linking individual preferences and societal goals is to strengthen public service media to make them diverse as well as attractive (Helberger 2015^A). Finding the balance between what consumers want and what leads to beneficial outcomes on the societal level is one of the key challenges of future media diversity research.

Finally, examining patterns of exposure while in itself being a valid goal, should for communication scientists only be the first step. In the end, the interesting question is to understand the consequences and effects that the exposure to different forms of content has. The idea of perceived or experienced diversity (Hoffman et al. 2015^A) should receive more attention here, stressing that only behavioral patterns are not enough to understand what ends up at the user.

In conclusion, this article has shown that diversity is an increasingly important concept to understand the contemporary news landscape and its individual and democratic effects. However, it also showed that urgent work is required both at the conceptual and the empirical level to revitalize this concept and adapt it to the radically different world of digital journalism in an era of audience fragmentation, social media, and news algorithms. This will require active collaboration between the fields surveyed in this article, with computer science, journalism and communication science, and legal studies and political philosophy each providing valuable input on how to define, operationalize, and benchmark the different sub-dimensions of diversity in the chain from news supply and (personalized) exposure to individual news diets.

Disclosure Statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by the NWO and the eScience center, Joint eScience and Data Science grant (DTEC.2017.014).

Notes

1 In the codebook this was operationalized as “Is an explicit definition of diversity given (indicated by phrases such as “diversity in this study is defined as…” or “diversity in this study consists of…”)?” If yes, all references made in the definition were coded as well as the full text of the definition.

2 The studies that are part of the literature review are referred to in the text as citation with an additional A as superscript to indicate that they can be found in Appendix A.

3 A full text search with a 20-word window around “democracy,” “democratic,” or “normative” in all articles, not including their reference section.

Appendix A Sources Used in Literature Review

Appendix B Results of literature review