Quality User-Generated Content? A Case Study of the Quality of Online News Comments on the Site of Finnish Public Service Broadcaster Yle

ABSTRACT

Over the years, studies have assessed the quality of discourse in comment sections and results have been mixed. Some argue that newspapers have created forums for hate speech against minorities, that the quality of discourse undermines the democratic potential of comment sections, or that comments are troubled by incivility. Others have found that commenters produce content resembling public deliberation and that comment sections can contain fruitful discussions on divisive topics. This article contributes to the literature on the deliberative quality of news comments by assessing the deliberative quality of news comments in Finland. What is the deliberative quality of comments on Svenska Yle’s news articles? We conduct a quantitative content analysis of 565 news comments to 30 articles on Svenska Yle. Using a novel measurement strategy derived and developed from earlier research on assessments of deliberative quality, we analyze the deliberativeness, emotiveness, and confrontativeness of comments. Our findings are relatively positive since the overall quality level exceeds that of several previous studies. However, several indicators of quality still need improvement.

KEYWORDS:

• News comments
• deliberation
• deliberative quality
• public sphere
• public service
• deliberativeness
• emotiveness
• confrontativeness

Introduction

A typical trait of online communication is the contrast between user-generated online comments and gatekeeper-provided news content produced by journalists (Lee, Atkinson, and Sung 2022). Since their introduction in the late 1990s (Santana 2011), news comments have offered a wider range of views compared to letters to the editor, possibly due to lower thresholds for commenting (McCluskey and Hmielowski 2012). Over time, news comments have become an integral part of online journalism (Beckert and Ziegele 2020, 3924), although problems such as incivility in comment sections have resulted in some news organizations abandoning comment sections entirely (Chen 2017). Either way, user-generated content (e.g., comment sections) is a central feature in the digital era and illustrates the changing role of public service broadcasting (PBS) organizations (Sjøvaag, Moe, and Stavelin 2012). The emergence of a hybrid media system is accompanied by increased public communication online and a decreased influence for journalistic and political elites (Pfetsch 2020).

In theory, digital public discourse in the form of news comments represents an ideal public sphere (McDermott 2016, 16). News comments have higher visibility compared to comments on independent forums and their unrestricted and immediate character seems to suit informal, non-hierarchical ways to participate in public dialogue (Zimmermann 2014, 100). News comments provide news organization with free user-generated content, which the audience comes to read. Comments offer citizens the ability to debate, discuss and provide views not present in the original article (Eisinger 2011, 4). High-quality discussion in comment sections might increase citizens’ understanding of political issues and thus alleviate ideological and affective polarization (Oswald 2022, X). The quality of news comments is important since news organizations want to host valuable exchanges of ideas while being seen as credible sources of information among the public (Diakopoulos and Naaman 2011, 133).

Yet, in practice, comment sections have often failed to live up to the ideals of a public sphere. Studies have found that comments are non-dialogical in nature (Parra, Ayerdi, and Fernández 2020), contain incivility (Chen 2017; Coe, Kenski, and Rains 2014), and demonstrate a low deliberative quality (Zimmermann 2014). Low-quality discussion causes several problems. Firstly, comment sections require moderation and response from news organizations, which takes up resources (Ihlebæk and Ytreberg 2009). Moderation expands the journalistic role and adds to journalists’ workload (Goodman and Cherubini 2013), while journalists often receive little training in moderation (Chen 2017). Secondly, low-quality comment sections might damage news organizations’ brands (Singer and Ashman 2009) or lower their credibility (Conlin and Roberts 2016). Thirdly, uncivil discussion recoil people from news sites (Diakopoulos and Naaman 2011; Springer, Engelmann, and Pfaffinger 2015).

Given the problems with the quality of news comments, it is unsurprising that news organizations are still experimenting with these. News outlets have tested various measures to improve the dialogue in the comment sections (Ksiazek 2015, 2018). For example, interactive moderation by journalists (Ziegele et al. 2018) and engagement moderation (Masullo, Riedl, and Huang 2022) have shown promising results. It seems likely that the experience news organizations have gathered over time should improve the quality of discourse. Research on the quality of comments has produced mixed results. Some argue that newspapers occasionally have created forums for hate speech against minorities (Santana 2012), that the quality of discourse undermines the democratic potential of comment sections (Nagar 2011, 147) or that comments are troubled by incivility (Coe, Kenski, and Rains 2014). Others have found that commenters produce content that resembles public deliberation (Manosevitch and Walker 2009) and that comment sections can contain fruitful discussions even on divisive topics (Friedman 2011).

This article contributes to the literature on the deliberative quality of news comments. To improve the discourse quality, scholars need to determine the baseline before any attempt to introduce a design feature for improving it. However, it is unknown how effective current strategies for eliminating dark participation, defined as “comments that transgress norms of politeness or honesty with partially sinister motives”, are (Frischlich, Boberg, and Quandt 2019, 2014). To assess the deliberative quality of online news comments in Finland, this article addresses the following research question: What is the deliberative quality of comments on Svenska Yle’s news articles?

To answer this question, we conduct a quantitative content analysis of 565 news comments to 30 news articles on Svenska Yle. We argue that Svenska Yle is a case of a national public-service broadcaster struggling with how to handle user-generated content. For example, Svenska Yle receives around 170,000 comments in a year, and these are moderated by reporters in addition to their ordinary journalistic work (Jungar 2018). Moreover, commenters constantly criticize the moderation (Jungar 2017). Public service media (PSM) are particularly interesting to study due to their perceived role as upholders of a forum for debate akin to part of a public sphere (see Fuchs 2014; Garnham 1990, 111), a role that has become accentuated in the digital era when PSM, ideally, take on a role as Public Service Internet (e.g., Fuchs 2014; Fuchs and Unterberger 2021). Employing a novel measurement strategy derived and developed from assessments of deliberative quality, we analyze the deliberativeness, emotiveness, and confrontativeness of comments.

Background and Theory

Comment Sections in News Outlets

News comments are a form of user-generated content in conjunction with online news articles (Santana 2011). Comment sections are relatively popular, many news organizations host these and people both post and read comments (Santana and Hopp 2022, 2). News comments are one of the most widespread types of user-generated content (Naab and Küchler 2022, 441). News comments are polylogues since they involve multiple levels of dialogue; between journalists/editors and the public as well as between the readers (Kolhaktar et al. 2020). Thus, comments can be both reactive and interactive (Ziegele et al. 2020, 865). In a historical perspective, news comments add a new dimension to traditional reporting and news readership as they offer readers a direct voice (von Sikorski and Hänelt 2016, 2).

News comments have many negative connotations. The mere presence of a comment section can lower the perceived quality of an article (Prochazka, Weber, and Schweiger 2018) or lower the credibility of news sites (Conlin and Roberts 2016). Journalists tend to dislike comments (Santana 2011) and regard them as abusive and of low quality (Mitchelstein 2011). Uncivil commenting concerns journalists, as they need direction on how to control commenting through engagement and moderation (Masullo Chen and Lu 2017). Nielsen (2012), however, argues that journalists can accept comment sections as long as they are not unmoderated spaces free-for-all. Journalists struggle to find balance between moderating comments for problematic behavior while maintaining an open discussion (Wolfgang 2018). A study on comment moderation strategies (Boberg et al. 2018) found that the moderator’s moral compass shape moderation rather than a set standard for gatekeeping. Wolfgang, McConnell, and Blackburn (2020, 941) conclude that news organizations need to allocate more resources to comment moderation instead of opening up comment sections without the possibility to manage these or outsourcing the discourse to social media platforms. Moreover, Wolfgang, McConnell, and Blackburn (2020) argue that journalism education and training should build better relationships with the audience by focusing more on audience engagement.

News comments can have a systematic influence on online journalism by affecting recipients’ processing of news (von Sikorski and Hänelt 2016). For example, comments function as peer influence (Chung 2019). Studies have found that comments have effects on readers’ opinions (Lee and Jang 2010; Rösner and Krämer 2016), although others (e.g., Steinfeld, Samuel-Azran, and Lev-On 2016) have found the opposite. Moreover, hateful or negative comments can lead to less charitable behavior (Weber et al. 2020). Masullo Chen and Lu (2017) found that disagreement in comments tended to have a chilling effect on public discourse. Although news comments are not representative of public opinion (Newman et al. 2017), they are used to assess the public debate (Naab and Küchler 2022, 442).

Why should news outlets aim to raise the quality of comments? Engelke (2020) shows that media users value and recognize deliberative elements and argues that news comments of deliberative nature are a reason for participation and positive evaluations. Other studies have indicated that raising the quality of comments might encourage users that otherwise stay silent to participate (Springer, Engelmann, and Pfaffinger 2015). Or, as Oswald (2022, 1) argues, high-quality comment sections can help users to understand issues and help to alleviate mechanisms of opinion polarization. Moreover, as comments have been found to influence peoples’ opinions (e.g., von Sikorski and Hänelt 2016; Ziegele et al. 2018) and behavior (e.g., Weber et al. 2020), high-quality comments might be contagious (Wolfgang and Bhandari 2020).

The (Deliberative) Quality of News Comments

During the last 15 years, scholars have discussed, analyzed, praised, and criticized comment sections (Ksiazek and Springer 2018). Criticism has sometimes led to comment sections being taken down altogether by news outlets such as Reuters and Bloomberg (Quandt 2018, 37). Ksiazek and Springer (2018, 482) argue that the quality of comment sections is the most pressing issue and see two solutions: news organizations either take ownership of comment sections or shut them down. Furthermore, they suggest, news organizations introduced comment sections to generate revenue and build user engagement and loyalty rather than to “curate public discussions” (Ksiazek and Springer 2018, 482). Shutting down comment sections can result in discussions moving to social media sites instead (Quandt 2018). However, this does not necessarily mean that discussions hosted on social media are of any better quality (Rowe 2015).

A common denominator in research about the quality of news comments is that scholars often evaluate findings against the background of deliberative norms such as rationality, inclusion, respect, and civility (Engelke 2020; Ziegele et al. 2020). Thus, studies on the quality of news comments have tended to have a normative approach influenced by deliberative democracy theorists (see Steenbergen et al. 2003). Deliberation here refers to public discussion between free and equal citizens (e.g., Fung 2003; Habermas 1996). Thus, deliberation is to be seen as a process-focused view on democracy, where discussion of sufficient quality is considered essential for a well-functioning democracy (Dahl 1989; Habermas 1996). Generally, studies on the quality of comments have tried to capture the multi-dimensional concept deliberative quality (e.g., Graham and Wright 2015). Most studies operationalize deliberative criteria and conduct quantitative content analyzes to measure the extent to which comments meet these criteria (e.g., Beckert and Ziegele 2020; see Naab and Küchler 2022). While Oswald (2022) has found that automated coding of deliberative quality can be useful and can complement manual content analyzes, it remains challenging to measure the deliberative quality of online discussions (Beauchamp 2020).

On a positive note, findings point out elements of deliberative discussions in comment sections (Rowe 2015; Ruiz et al. 2011; Strandberg and Berg 2013; Yang 2022). For example, comment sections can display rationality (Esau, Friess, and Eilders 2017; Graham and Wright 2015), inclusion (Esau, Friess, and Eilders 2017), and civility (Ksiazek 2015). Moreover, comments can reach acceptable levels of respect (Manosevitch and Walker 2009; Ruiz et al. 2011; Strandberg and Berg 2013), relevance (Collins and Nerlich 2015), civility (Eisinger 2011; Friess, Ziegele, and Heinbach 2021; Zhou, Chan, and Peng 2008), and reciprocity/interaction (Esau, Friess, and Eilders 2017; Graham and Wright 2015).

On a negative note, previous research has found several problems. Noci et al. (2012) conclude that comments do not foster a democratic dialogue. Others similarly find that comments fail to live up to their deliberative potential, constitute low-quality discussion (Zimmermann 2014), and lack a dialogical nature (Parra, Ayerdi, and Fernández 2020). Other problems include incivility (Anderson et al. 2014; Chen 2017), racism (Harlow 2015), hate speech (Erjavec and Kovačič 2012) impoliteness (Neurauter-Kessels 2011), and trolling (Binns 2012). Santana (2019) found low levels of rationality and high levels of uncivility in both anonymous and non-anonymous comments. Likewise, an early study by Paskin (2010) indicated that comment sections feature off-topic conversations and personal attacks while simultaneously showing high levels of interaction between readers. Furthermore, research focusing on perceived quality of comments has found that the level of interaction does not seem to live up to deliberative ideals (Diakopoulos and Naaman 2011).

Findings about the quality of news comments have been inconclusive, although it is easier to point out pessimistic than optimistic conclusions (Friess, Ziegele, and Heinbach 2021, 627). Ziegele et al. (2020, 861) conclude that the “pessimistic perspective of user comments seems to have gained the upper hand.” Mixed findings regarding quality have resulted in researchers turning their attention toward factors affecting the deliberative quality of news comments. Marzinkowski and Engelmann (2022, 433) and Stroud et al. (2015) conclude that decisions by news organizations as well as users’ argumentation both relate to the quality of discussions. Strategies for improving the quality of discussion include: moderation (Masullo, Riedl, and Huang 2022), user registration (Moore et al. 2021), comment policies (Ksiazek and Springer 2018), journalists/editors engaging in conversations (Stroud et al. 2015), selectively opening up stories for commenting (Nelson, Ksiazek, and Springer 2021), adding possibilities to highlight or upvote comments (Kolhatkar and Taboada 2017), and different mechanism for sorting comments (Peacock, Scacco, and Jomini Stroud 2019). Variations of these strategies are all in use in the public broadcaster comment sections this study analyzes.

Measurement and Data

The Context of the Study

Our study focuses on the Finnish public service broadcaster Yle’s news comments. Public service media are particularly interesting to study in this regard due to their perceived role as upholders of a forum for debate akin to part of a public sphere (see Fuchs 2014; Garnham 1990, 111). Yle states in its policies: “Yle supports democracy and citizens’ opportunities to participate through diverse discussions in the services and social media accounts maintained by Yle” (Yle 2023). In fact, Yle’s digital policies are eerily similar to the Public Service Internet-manifesto on how PSM should ideally perform their role in the digital age (see Fuchs and Unterberger 2021). Furthermore, Yle has an important role in Finnish society in general (Horowitz and Leino 2020), being one of the most trusted media outlets with over 90 percent of Finns feeling that Yle is reliable or very reliable (see Matikainen et al. 2020). Furthermore, Yle has a large audience with 96 percent of Finns accessing at least one of Yle’s services on weekly basis. Yle’s audience is diverse in terms of demographics and political views (see Schulz, Levy, and Nielsen 2019, 23). Yle clearly has an important role in Finnish society which makes it crucial that the public discussions it upholds are of good quality.

The data in the study derives from readers’ comments on online news stories published by Svenska Yle, the Swedish language branch Yle. With its high internet penetration rate and high overall trust in online media (Horowitz and Leino 2020), Finland can be considered a typical instance of a highly digitalized media environment. Finland is also a representative of the democratic corporatist media system (see Hallin and Mancini 2004 for description of traits of such media systems). We argue that Svenska Yle is a case of a national public broadcaster struggling with how to design, handle, and moderate comment sections (Jungar 2017, 2018). The main features Svenska Yle uses to ensure quality commenting are pre-moderation by journalists/editors, pseudonymous commenting (user registration), and journalistic engagement in comment sections (Yle 2023). This is exactly the same setting as the larger Finnish-speaking section of Yle uses. In an international comparison (cf., BBC), however, Svenska Yle is a non-typical PSM as its main target group is the Swedish-speaking minority, which consists of about five percent of the Finnish population. We regard Svenska Yle as a most-likely case for finding deliberative quality in comment sections due to the following reasons. First, commenting on Svenska Yle is pseudonymous, a setting producing more high-quality commenting than both anonymous and real-name commenting (Moore et al. 2021; see also Rowe 2015). Second, Svenska Yle should have more resources for comment moderation compared to smaller local news outlets (cf., Su et al. 2018, 3683). Arguably, as a public service broadcaster, Svenska Yle has a larger responsibility to host impartial democratic debates and to present diverging perspectives compared to private news outlets (Hallin and Mancini 2004). Thus, Svenska Yle should not host comment sections for the sake of business strengthening or customer loyalty, but rather for providing an arena for public debate and for performing their role as a public service internet (see Fuchs and Unterberger 2021; Quandt 2018, 38). However, PSM do not operate completely outside of market logics, as they have to adapt to competitors use of online platforms by enabling, for example popular features such as algorithmic recommendations, which may bias the exposure of diversity, a fundamental value of PSM (Sørensen and Hutchinson 2018). Today, PSM compete for attention with social media such as Youtube, Tiktok and streaming platforms such as Netflix and Spotify, perhaps prompting PSM to shape user interaction promoting business interests rather than fostering democracy (Van Es and Poell 2020, 1). It seems as PSM are under pressure to extend their audience reach by employing functions found in other social media platforms to not be left behind (Van Es and Poell 2020, 2–3).

Measuring Discourse Quality

Several attempts to measure deliberative quality exist (e.g., Esau, Friess, and Eilders 2017; Marzinkowski and Engelmann 2022; Ziegele et al. 2020). A well-known contribution is the Discourse Quality Index (DQI), developed by Steenbergen et al. (2003). Relying on the Habermasian theory of communicative action, DQI understands deliberation as “a systematic process wherein actors tell the truth, justify their positions extensively […] and are willing to yield to the force of the better argument", with the ultimate goal of reaching understanding, or consensus (Bächtiger et al. 2010, 33).

This prototypical view of deliberation has, however, been widely challenged (see Bächtiger et al. 2018 for an overview), with critics characterizing classic deliberative theory (and, by implication, DQI) as too idealistic (e.g., Shapiro 1999), too consensus-oriented (e.g., Mouffe 1999), too demanding (e.g., Posner 2004) and too rational (e.g., Sanders 1997). Consequently, contemporary deliberation research has departed somewhat from a narrow focus on rational discourse (e.g., Marzinkowski and Engelmann 2022; see Rossini and Stromer-Galley 2019, 6). Bächtiger and Parkinson (2019) admit that DQI is insensitive to diverging goals and contexts. “Actors not only argue”, they claim, but also “use rhetorical strategies”, “tell stories or anecdotes”, “use confrontational or sarcastic language” and “bargain and make promises or threaths” (Bächtiger and Parkinson 2019, 70). In their view, future research should take this into account and incorporate a greater variety of communication forms into the deliberative toolbox, however, without eliminating the distinction between the deliberative core on the one hand, and other forms of communication on the other.

Our measurement strategy builds on these observations. Our intention is, hence, to capture the core components of deliberation and to examine the occurrence of other purportedly common discourse strategies such as the use of emotive and confrontative language.

Measuring Deliberativeness, Emotiveness and Confrontativeness

Building on Goertz (2006, 2020), we structure our conceptualization and measurement in three levels. At the basic level, we find our core theoretical concepts deliberativeness, emotiveness and confrontativeness. The secondary level provides the constitutive dimensions (i.e., core definitions) the basic level concepts while the indicator level operationalizes the dimensions, allowing us to measure whether some specific phenomenon is an instance of that dimension or not.

Our first core theoretical concept is deliberativeness. Based on recent revisions of the prototypical model of deliberative theory (Cohen 1989; Habermas 1981), the term can be minimally defined by its two core components—reason-giving and listening (Bächtiger and Parkinson 2019, 23–24, 105–106). Reason-giving is the linking of claims to reasons. In this study, it is measured using four indicators, focusing, respectively, on the existence and level of argumentation, on external and internal validity, and substantive relevance. Listening, in contrast, refers to weighing and reflecting on perspectives given by others. To measure listening we use two indicators, capturing reciprocity (interaction with others) and openness (receptivity to other viewpoints), respectively.

As noted above, some deliberative theorists have suggested that alternative, not strictly rational, forms of communication should be considered in empirical studies of deliberation. Dryzek (2000, 1), for example, proposes the inclusion of “rhetoric, humour, emotion, testimony or storytelling […] and gossip” but rules out more rebellious elements such as manipulation, deception, and threats. Young, by contrast, emphasizes how different backgrounds, experiences, and status might affect discursive abilities. Democratic deliberation should, therefore, not be restricted to “polite, orderly, dispassionate, gentlemanly argument”, but also allow “[d]isorderly, disruptive, annoying, or distracting means of communication” (Young 2000, 49–50).

Considering these remarks, this article empirically studies two additional forms of communication—emotiveness and confrontativeness. Emotiveness refers to core forms of passionate communication such as expression of and appeal to emotions (Bächtiger et al. 2010, 43; Bächtiger and Parkinson 2019, 70; Dryzek 2000, 1, 167; see empirical analyzes by Esau, Fleuß, and Nienhaus 2021; Marzinkowski and Engelmann 2022; Noci et al. 2012). To measure emotional expressions, we use indicators capturing manifest expressions of positive and negative emotions. Appeal to emotions is measured using indicators focusing on storytelling, i.e., the sharing of personal experiences and the telling of figurative anecdotes.

Confrontativeness, finally, captures communication that does not follow ideal-typical deliberative virtues such as politeness and civility, instead including both polemic antagonism and acerbic provocation (Bächtiger et al. 2010, 43; Bächtiger and Parkinson 2019, 70; Young 2000, 49; see empirical analyzes by Marzinkowski and Engelmann 2022; Strandberg et al. 2021). The indicators used to measure antagonism focus on disagreement (on substantial matters) and disapproval (of others’ cognitive capabilities). Provocations, in turn, are measured by the two indicators incivility and rigidity.

A more detailed description of the coding scheme is given in Online Appendix A. To test its reliability, three rounds of reliability tests were performed. After each round, coding instructions were revised, with more details (including coding examples) provided to coders. After the second round, the variables disagreement, disapproval and incivility were separated into three categories each, focusing on other debaters, journalists and publishers, and interviewees, respectively. Table 1 reports inter-coder reliability scores for all variables.

Table 1. Reliability test of coding scheme.

Basic level concept	Dimensions	Variable	PA	K-α
Confrontativeness	Reasoning	Argumentation	74	0.573
		Internal validation	71	0.238
		External validation	90	0.747
		Relevance	66	0.463
	Listening	Reciprocity	84	0.737
		Openness	95	0.724
Emotiveness	Emotional expressions	Positive emotions	92	0.366
		Negative emotions	92	0.725
	Storytelling	Personal experiences	97	0.789
		Anecdotes	95	0.643
Confrontativeness	Opposition	Disagreement 1	87	0.714
		Disagreement 2	97	0.658
		Disagreement 3	97	0.845
		Disapproval 1	97	0.658
		Disapproval 2^a	100
		Disapproval 3	100	1
	Provocation	Incivility 1	97	0.000
		Incivility 2	97	0.000
		Incivility 3	90	0.518
		Rigidity^a	100

Inter-coder reliability was tested using three (first round) or two (second and third round) coders on a random subsample of 38 comments. Reliability is reported as percent agreement (PA) and Krippendorff’s alpha (K-α) (see Lacy et al. 2015; Lombard et al. 2017). Results are from the final third round of training and coding instruction revision. One coder conducted the coding of the full sample of 565 comments.

^a Krippendorff’s alpha undefined due to no variation among coders (both coded 0 on all comments).

The Data

The data was collected from 1.11.2020 to 31.10.2021.¹ To ensure that comments were part of longer discussions we excluded news stories with less than ten comments. We also excluded genuine sports news (e.g., match reports) and announcements related to Svenska Yle’s own radio programs. To keep the number of comments manageable, we adopted a two-step random selection strategy. First, to cover news stories from all five regional editorial offices of Svenska Yle (Åboland, Capital region, Österbotten, Östra Nyland and Västra Nyland) each region was given two randomly selected months. The remaining two-month period (60 days) was divided into five equally sized parts, with each region being randomly given one part (i.e., one 12-day period). In the second step, we used stratified random sampling to select 30 (of the in total 157) news stories for further analysis. Our data set includes 565 news comments. Of these, 22 (3.9 percent) were written by an Yle journalist. The number of unique pseudonyms was 270, making the average of unique comments per pseudonym 2.1. The average length of comments was 294 characters. Tables A1 and A2 in the Online Appendix provide more detailed descriptive information on news and comments.

Results

To what extent does pre-moderated, pseudonymous commenting on Svenska Yle’s news live up to deliberative ideals of reasoning and listening? Moreover, to what extent are elements of emotiveness and confrontativeness present?

Figure 1 reports on reason-giving. As Figure 1(a) shows, a clear majority of comments (77 percent) contain some form of argumentation. Incomplete argumentation occurs, however, more frequently than qualified argumentation: only about one-third of comments providing argumentation in favor of the advanced position also try to illuminate the linkage between the argument and the position. Figure 1(b) illustrates the relevance of comments. As is evident, most comments (78 percent) are directly relevant to the topic of discussion, and only a few (9, or 2 percent) are entirely irrelevant. Figure 1(c) demonstrates the use of internal and external validation. Most commonly, comments use internal validation—nearly half (47 percent) of the comments validate the preferred position by references to commentators’ values, opinions, and beliefs, without citing any external sources (e.g., facts or figures). 17 percent use only external sources while 12 percent use both external and internal sources. A sizeable minority of comments—23 percent—include neither external nor internal validation.

Figure 1. Reason-giving.

In Figure 2, we focus on the second core feature of deliberativeness—listening. As Figure 2(a) shows, a typical comment does not interact with other comments. Moreover, it seems that spontaneous reciprocity (30 percent) is more common than contemplative reciprocity (27 percent). A striking observation is that most comments seem to be rather firm. As shown in Figure 2(b), most commentators (89 percent) tend to cling to their own viewpoints, without showing any appreciation or recognition of views presented by others. Reconsideration is even rarer than appreciation; only one single comment shows explicit signs of being persuaded by argumentation given in another comment.

Figure 2. Listening.

*  *  *

Next, we move to emotiveness and confrontativeness. Regarding emotive content, the results clearly indicates that overt expressions of emotions are rare. Few comments show unambiguous evidence of negative emotions, and even fewer include positive emotions (4 and 2 percent, respectively). Similarly, very few comments appeal to emotions. Only 3 percent share stories of personal experiences and no more than 1 percent tell anecdotes. These observations point, above all, to the elusive nature of emotional content: systematic capturing of emotions from textual sources is a challenging, if not impossible, task. We opted to only include manifest expressions of emotions as an indication of emotiveness. There is thus a possibility that there are implicit expressions of emotions that are here treated coded as non-emotive. Therefore, our observations are clearly of tentative nature, emphasizing the need for more research on the use of emotions in news commenting.

Turning to confrontativeness, we begin by examining to what extent comments convey opposition, that is, disagreement with others on substantial matters and disapproval of other’s cognitive capabilities. As Table 2 shows, disagreement with other debaters occur with some frequency. About one third (35 percent) of all comments express total disagreement with at least one other debater. Partial disagreement is less frequent, with only 4 percent of comments expressing both agreement and disagreement with other debaters. Table 2 further reveals that disagreement—to the extent that it occurs—is oriented towards other debaters rather than towards Svenska Yle (journalists/editors) or interviewees. The same general pattern occurs in Table 3; disapproval is, undoubtedly, less common than disagreement, but when it occurs it is typically directed towards other debaters, and only rarely towards Svenska Yle and/or towards interviewees.

Table 2. Disagreement.

	No disagreement		Partial disagreement		Total disagreement
	n	%	n	%	n	%
… with other debaters	344	60.9	25	4.4	196	34.7
… with Yle	545	96.5	2	0.4	18	3.2
… with interviewees^a	368	96.3	1	0.3	13	3.4

^a Comments to articles without interviewees excluded.

Table 3. Disapproval.

	No disapproval		Disapproval
	n	%	n	%
… of other debaters	533	94.3	32	5.7
… of Yle	558	98.8	7	1.2
… of interviewees^a	378	99.0	4	1.0

^a Comments to articles without interviewees excluded.

The second element of confrontativeness, provocation, is virtually non-existent in comments. Nearly all debaters express their views in a civilized manner, without reverting to stereotyping or harassment. Rigidity is also very rare, with only eight comments including conspiratorial reasoning (and no comment including subversiveness). Hence, the study of confrontativeness leads to two major conclusions. First: opposition is rare, but more common than provocation. Second: it is more common to challenge other debaters’ viewpoints (disagreement) than to question their cognitive capabilities (dislike).

Discussion and Conclusions

News comments are somewhat of a two-edged sword. On the one hand, they hold the potential for constituting a digital public sphere full of rational, reciprocal and democratic discussion between citizens. On the other hand, though, the reality of online comments has often turned out to be rather harsh (e.g., Coe, Kenski, and Rains 2014; Parra, Ayerdi, and Fernández 2020; Santana 2019), which has typically led to especially commercial media closing comment sections altogether (Graf 2021; Quandt 2018, 37; Ziegele et al. 2020, 861). Often-occurring malaises of news comments have been a general lack of argumentation and fact-based discussion (e.g., Santana 2019), a high degree of emotionally driven messages and confrontations between online commentators (e.g., Coe, Kenski, and Rains 2014; Erjavec and Kovačič 2012). We explored these elements in news comments on the Finnish public service broadcaster Yle’s website. Public service media, we argued, hold a central position in the surrounding society and hold an obligation to uphold the societal discussion (see also Yle’s guidelines for discussion, Yle 2023). The purpose of this article has been to assess the deliberative quality of the public sphere in Finland by addressing the research question: What is the deliberative quality of comments on Svenska Yle’s news articles? Using a systematic quantitative content analysis of 565 news comments, we analyzed their deliberativeness, emotiveness, and confrontativeness. What were the main findings of our exploration of Svenska Yle’s news comments?

Starting with deliberativeness, which measures the amount of reasoning and listening commenters practice, our findings are mixed. On a positive note, commenters mostly try to argue their points implying that comments usually are rational, which is in line with previous findings (e.g., Esau, Friess, and Eilders 2017; Graham and Wright 2015; Yang 2022) while contradicting others (e.g., Noci et al. 2012; Santana 2019). Our findings resemble those of Zhou, Chan, and Peng (2008) in that comments featured argumentation, but that the complexity of argumentation was relatively low in the sense that commenters could use external sources more often and provide clearer connections between their positions and reasons for these. Related to the complexity of argumentation, commenters are more prone to use internal validation than external validation. They mainly refer to their own values, opinions and information when making an argument. Still, external sources and information a found in 17 percent of comments, which is a higher share than findings in similar studies (Noci 2012; Parra, Ayerdi, and Fernández 2020; Rowe 2015; Strandberg and Berg 2013). Thus, similar to findings by Manosevitch and Walker (2009), commenters on Svenska Yle seem relatively prone to share additional information in their comments, indicating that their comments add value to discussions.

Regarding relevance, that is, commenting on the topic of the article, the results show that irrelevant comments are very rare, similar to previous studies (Collins and Nerlich 2015; Rowe 2015; Ruiz et al. 2011; Strandberg and Berg 2013). This finding is the opposite of other studies where a vast majority of comments was irrelevant (Coe, Kenski, and Rains 2014; Parra, Ayerdi, and Fernández 2020). The pre-moderation of comments might be the explanation for the high share of on-topic comments (Ruiz et al. 2011).

Another aspect of deliberativeness is the amount of listening between commenters (Scudder 2022). Therefore, commenters should show reciprocity with others and be open to each other’s viewpoints (Friess, Ziegele, and Heinbach 2021). Here, our findings leave room for improvement, as the most common type of reciprocity is no reciprocity at all. However, commenters do reply to each other, albeit mostly in a spontaneous manner. The share of reciprocal comments in Svenska Yle’s comment sections is slightly higher than previous findings of Strandberg and Berg (2013) and over 20 percentage points higher than in the study of Zhou, Chan, and Peng (2008, 766). Yet, Esau, Friess, and Eilders (2017) and Esau and Friess (2022) found a higher share, 76 percent, of reciprocal news comments in German news media. Thus, in a relative perspective, the reciprocity in this study was acceptable and quite similar to previous studies (Graham and Wright 2015; Santana 2019).

As emotions, personal experiences, and humor are essential parts of online discussion (Rossini and Stromer-Galley 2019, 6), we sought capture the concept emotiveness by measuring expressions of emotions and appeals to emotions in comments. By doing so, we investigate the “tolerant concept of deliberation” focusing on expressions of emotions and storytelling in addition to the “classic concept of deliberation” focusing on rationality, reciprocity, respect, and constructiveness (Esau, Fleuß, and Nienhaus 2021, 10). In sum, we found emotional expressions to be very rare, 4 percent of comments featured negative emotions and only 2 percent positive emotions. This is a much smaller share compared to previous research on emotional expression in news comments. For example, Santana (2019) found emotional expressions in 28–46 percent of comments, Esau, Fleuß, and Nienhaus (2021) detected 2 percent positive and 23 percent negative emotions, while Marzinkowski and Engelmann (2022) found 15 percent negative emotions. This finding is rather surprising, given that studies have found high levels of negative emotions in news comments (e.g., Anderson et al. 2014). However, as we alluded to in the results-section, our restrictive coding could be one explanation for the low share of emotions in our data. Another possible explanation could be that all our analyzed comments were moderated, and emotive content might have been omitted by Yle before publication.

The concept confrontativeness means communication recognized as both polemic antagonism and acerbic provocation rather than ideal-typical deliberative virtues such as politeness and civility. To capture this type of speech, we measured the level of disagreement, disapproval, incivility, and rigidity. Scholars measure the level of disagreement in discussions because some level of disagreement is necessary to signal that different viewpoints are present (Esterling, Fung, and Lee 2015; Gastil 2018). Online discussions lacking disagreement might polarize opinions due to like-minded people discussing with each other (Wright, Graham, and Jackson 2017). We found disagreement in over one third of the comments, while disapproval was very rare; incivility and rigidity were practically non-existent. Previous studies of news comments have found varying levels of disagreement. The level of disagreement we found is similar to previous findings (Strandberg and Berg 2013; Zhou, Chan, and Peng 2008), but lower than others (Santana 2019). Disagreement in news comments in not necessarily a problem as long as it does not digress into incivility (Masullo Chen and Lu 2017) and comments with high argument strength tend to receive (reasoned) disagreement (Marzinkowski and Engelmann 2022). Thus, our results can be considered quite positive, as there is moderate disagreement in the comment section (Esterling, Fung, and Lee 2015). However, it is difficult to set a threshold for how much disagreement is healthy in comment sections. Finally, several studies on news comments have found high levels of incivility (Chen 2017; Coe, Kenski, and Rains 2014; Parra, Ayerdi, and Fernández 2020), while, in contrast, our study found very low levels of incivility. Therefore, our findings are in line with positive findings (Eisinger 2011; Strandberg and Berg 2013; Zhou, Chan, and Peng 2008).

Some limitations of our study are worth mentioning. Did an effective moderation strategy ensure quality in the comments? We cannot say much about the causal relationship between the measures (e.g., moderation, journalistic involvement) Svenska Yle has taken and the quality of deliberation. An answer to these types of questions would require an experimental design. We asked Svenska Yle for access to deleted/refused comments to investigate their deliberative quality. Our request was denied. Moderators refuse a considerable share, 26–30 percent, of comments (Hindsberg 2016). Due to the pseudonymity allowed at Svenska Yle, we cannot analyze the gender distribution among commenters. Gender equality is a measure of quality related to inclusion and a previous study of BBC’s online forums found high gender inequality as men dominated the forum (Quinlan, Shephard, and Paterson 2015). As previously noted, we found it difficult to achieve high inter-reliability measured with Krippendorff’s alpha for some variables although previous studies have reported high inter-coder reliability using similar instruments such as the discourse quality index (DQI) (Bächtiger, Gerber, and Fournier-Tombs 2022, 86; see also Beckert and Ziegele 2020, 3931). A final potential limitation could be that our case Svenska Yle, being a PSM for a language minority, might not be entirely representative of all PSM. Thus, our conclusions from Svenska Yle may not be readily generalizable per se albeit that, as we have discussed, Svenska Yle generally displays the same traits as PSM in general and public service internet as well.

To conclude, this study analyzed the deliberative quality of news comments on the website of Finnish public service broadcaster Svenska Yle. Doing so, we argue that we fill a research gap by studying news comments hosted by a public service broadcaster. In a comparative perspective, our findings are positive since the overall quality level exceeds that of several previous studies. As to why that is the case, we can only be speculative since we have not systematically tested factors that lead to high comment quality. Nonetheless, it is perceivable that the strict public service internet ethos of Svenska Yle (and Yle in general) manifested in their policies for moderation, is one explanation of the high quality we found. Of course, the mere fact that all comments were moderated serves to raise quality as well. A further potential contextual factor might be the Finnish public’s perception of Yle as a very high quality and trustworthy news outlet, which could induce people into being more constructive in their comments. Media systems with strong public service media, compared to other media systems, tend to display good resilience to malaises of the digital realm (see e.g., Fletcher, Cornia, and Nielsen 2019; Humprecht, Esser, and Van Aelst 2020). We believe that our findings are indicative of the same phenomenon. Another contribution of this study is also a novel coding scheme for measuring the quality of news comments. Moreover, we have discussed methodological lessons learnt from our exploration. In sum, our findings point away from pessimistic conclusions about the quality of news comments, as Svenska Yle’s comment sections do not feature dark participation and incivility. As public service media, and public service internet (Fuchs and Unterberger 2021) are central and trusted actors and arenas for upholding a public sphere (Fuchs 2014; Horowitz and Leino 2020), our findings are rather encouraging for the potential of digital public debate when conveyed through high-quality media platforms.

Disclosure Statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This research is a part of the project “The societal discussion climate—from polarization and hate speech to constructive and democratic dialogue” which is funded 2021–2024 by Högskolestiftelsen i Österbotten.

Notes

1 This time period was chosen to exclude late fall 2021, when Svenska Yle applied a new, stricter, policy regarding the opening of news stories to readers’ comments.