The cueing power of comments on social media: how disagreement in Facebook comments affects user engagement with news

ABSTRACT

Previous research demonstrates that conflict framing in news articles can influence individuals’ attention, selection, and distribution of news. However, no study has examined whether the valence of social media comment fields can trigger similar effects for news engagement on Facebook. In this mixed-methods study, we combine eye tracking with surveys, and conduct an experiment in which participants (n = 96) were exposed to 20 Facebook news posts from the Swedish tabloid Aftonbladet. Under each post, we presented participants with a pair of real (but anonymized) Facebook comments that were either in agreement or disagreement with one another. We then examined how this manipulation influenced participants’ visual attention to comment fields, their self-reported likelihood to click on the post to read the full story, and their self-reported likelihood to share the news post to their Facebook network. Our results show that comments in disagreement increased users’ visual attention to comments, decreased their likelihood to share the post, and had no effect on their likelihood to read the news article associated with the post. Thus, the presence of disagreement in comments does cue news engagement on Facebook, but the effect is not uniform across different news engagement behaviors. Moreover, engagement with hard versus soft news topics also varied. Disagreement in comments to Facebook posts about soft news topics (Entertainment, Society, and Sports) increased users’ attention to the comments field. In contrast, comment disagreement for hard news topics (Economy and Politics) reduced users’ attention to the comment field, as well as their self-reported likelihood to read the post.

KEYWORDS:

• Social media
• eye tracking
• Journalism
• news sharing
• social news gap
• visual attention

Introduction

Informed citizens are vital for a well-functioning democracy. Although social media platforms have multiplied the information channels available to citizens, the content served by algorithmically-driven systems may not always be meaningful in a civic sense. Social media platforms are ‘third spaces’ (Wright et al., 2015) where news about politics is intertwined with personal, entertainment, and lifestyle-oriented content. Some citizens overly rely on platforms to deliver news to them and have disengaged from news-seeking altogether (Gil de Zúñiga et al., 2017). Therefore, uncovering the mechanisms that guide citizens’ engagement with news on social media is an important task for communication research.

Prior studies suggest that on social media, certain journalistic frames can elicit online behaviors such as clicking news stories, commenting on their content, or sharing them across networks (Stroud & Muddiman, 2019; Valenzuela et al., 2017). However, the question of whether social media comments can trigger similar effects on news engagement has remained underexplored. This lack of knowledge regarding the role of comments in citizens’ news engagement on social media is problematic for three reasons. First, social media platforms are increasingly sites for daily news consumption. Second, the design of these platforms fosters interactivity, with comment fields placed directly below news posts. Because of their high visibility, comments as citizen-generated opinions may influence the news diets of their network peers. Third, as several online news portals remove comment sections or put paywalls in place, social media platforms are quickly establishing themselves as public fora for discussing a broad range of news. In this study, we conduct an experiment that isolates one dynamic of user commenting – the presence or absence or disagreement – and provide initial evidence that user-generated commentary influences the news engagement patterns of citizens on Facebook.

Theoretical approach

Unlike radio, television, and print, news stories on social media largely lack the editorial cues that have traditionally signaled the news value of a story. Graber (1988) found in newspapers, for example, that cues such as the location of a story on the page, its length, and the size of headlines and pictures influenced its chance of being read. However, most social media platforms present content via a standardized template, which limits journalists’ ability to signal the importance of a news story in these traditional ways. On Facebook’s News Feed, images are fitted to predefined pixel dimensions, headline sizes do not vary, and users cannot assess the length of a story without clicking on a post to read the full-length article. We are therefore interested in better understanding the mechanisms that influence users’ engagement with news on Facebook, where traditional editorial cues are largely absent.

Facebook users can signal the relevance of posts to the community through reactions, comments, and shares, which we collectively call ‘community cues.’ On social media, community cues can effectively supplant editorial cues, resulting in what Messing and Westwood (2014, p. 1045) refer to as the ‘socialization of the news consumption process.’ A small but growing body of experimental research has examined the extent to which community cues guide users’ interaction with news online (Dvir-Gvirsman, 2019; Messing & Westwood, 2014; Pierce et al., 2017; Sülflow et al., 2019; Winter, 2018; Winter et al., 2015). The findings of this literature reveal three key trends that motivate our research design.

First, the presence of community cues such as likes (Messing & Westwood, 2014) and comments (Dvir-Gvirsman, 2019) increases the probability of selecting a news story. However, these experiments compared stimuli bearing community cues with those completely lacking such indices – which is not a real-life situation but a laboratory setting. Second, comparisons between cue type – quantitative or qualitative – reveal that qualitative cues such as comments increase the likelihood of a post to be selected more than quantitative cues – e.g., the raw number of reactions or shares (Dvir-Gvirsman, 2019).

Third, the valence of cues influences news selection and evaluation. When experimental stimuli vary between positive and negative community cues, negative cues appear to exert a larger influence, regardless of cue type (quantitative or qualitative). Quantitative negative cues like ‘downvotes’ decrease news selection and evaluation of actors in the story (Pierce et al., 2017), whereas positive quantitative cues, such as Facebook ‘likes,’ have virtually no effect on news selection (Dvir-Gvirsman, 2019) or users’ evaluation of news content (Winter et al., 2015). Similarly, negative comments on Facebook have been shown to adversely affect users’ attitudes toward news items (Winter, 2018) and policy issues (Winter et al., 2015), whereas positive comments have no effect.

To summarize, the presence, type, and valence of community cue has been shown to influence users’ engagement with news on Facebook. Although the studies outlined above test the effects of different cues on various news engagement behaviors, a synthesis of their results suggests that the presence of negative qualitative cues (such as comments) are an influential driver of citizens’ decisions to select and evaluate news.

Research on news framing gives further evidence that conflict frames influence news engagement, but the direction of their influence is ambiguous. Trilling et al. (2017) demonstrate that conflict increases citizens’ likelihood to share news on Facebook, whereas Valenzuela et al. (2017) find the opposite. In both studies, conflict is operationalized as disagreement between individuals or groups. Thus, although evidence remains mixed regarding whether conflict frames increase or decrease sharing, the presence of disagreement appears to be an influential factor in citizens’ news engagement behavior. In sum, there is evidence that community cues below a news post can influence news engagement on social media, and that journalistic frames involving disagreement can influence sharing. We seek to test whether disagreement between comments can exert similar effects as those found in studies of journalistic frames.

Disagreement

In addition to its relevance in journalistic reporting, disagreement is a facet of civic life that is intrinsically connected to democracy. Citizens’ tolerance of disagreement is the essence of liberal democracy, because it confers legitimacy to conflict, which is expected to be managed by state institutions (Huckfeldt et al., 2004, pp. 2–7). At its core and as defined by the Oxford English Dictionary, disagreement is a ‘difference of opinion.’ However, the degree of disagreement can vary, and scholars have associated high levels of disagreement with concepts such as conflict (Trilling et al., 2017, p. 49; Valenzuela et al., 2017, p. 812) and controversy (Ziegele et al., 2018, p. 1420). Chen and Berger (2013, p. 581) echo the scalable nature of disagreement, suggesting that affective involvement in a difference of opinion constitutes controversy. These subtle nuances have a direct bearing on research. A study comparing responses to questions defining disagreement as either the absence of agreement or as strongly divergent preferences found that depending on which measure is employed, different results can be obtained (Klofstad et al., 2013).

To avoid conceptual slippage between disagreement and related concepts such as conflict and controversy, we conceptualize disagreement in its most basic form – as the difference of opinion. Empirically, scholars have approached disagreement in one of two ways: as the explicit expression of contrasting opinions (Huckfeldt et al., 2004) or as the perception of discrepancy between two opinions (Mutz, 2006). Here, we choose the first approach because it aligns with the medium we study. Prior research has shown that Facebook users are exposed to news items that run counter to their viewpoints (Bakshy et al., 2015), and comment fields offer an additional space for citizens to observe counter-attitudinal opinions and disagreement on the platform.

Given the influential role that disagreement-oriented journalistic frames appear to exert on users’ news engagement, dissenting comments may therefore be an overlooked driver of citizens’ interaction with news on social media. In the following sections, we clarify our understanding of social media news engagement and formulate hypotheses regarding the role of disagreeing comments on users’ attention, selection, and distribution of news on Facebook.

News engagement on social media

Depending on the architecture of a social media platform, users can interact with content through varying degrees of engagement (Bossetta et al., 2017). For our purposes here, the term ‘engagement’ constitutes three distinct, yet interrelated behaviors that influence news consumption and distribution on Facebook. When users are presented with an abundance of continuously updated information, such as with Facebook’s News Feed, they must first direct visual attention to (i.e., look at) a news post. Users can then choose to read the full news article through selection (i.e., clicking the post to read the full story). In addition, users can share the post with their network of peers, an online activity that contributes to the distribution of news on social media. These three behaviors do not constitute the entire range of possible engagement with news, but we focus on them to help structure our research design and the variables we test.

Attention

Attention is the observable step following news exposure and preceding news processing (Kruikemeier et al., 2018). Here, we define attention as the act of looking at a specific design element of a webpage.¹ While minimal attention to an element confirms that the viewer has been exposed to it, further aspects of gaze behavior can reveal more about the information-processing of the viewer (Duchowski, 2002). For instance, the amount of time spent looking at an element may reflect the extent of cognitive resources deployed to process that element, as posited by theories such as the eye-mind hypothesis (Just & Carpenter, 1980) and suggested by experiments on reading (Rayner, 1998). Furthermore, increased attention has been argued to play a decisive role in decision-making (Pärnamets et al., 2015). Previous research on internet users’ visual attention to online advertising has utilized the distinction between potential exposure to ads (i.e. ads were present on the web page), actual exposure (users’ gaze point coincided with advert area), and perceived exposure (the user recalled having seen an advert that may or may not have been looked at during website interaction) (Gidlöf et al, 2012).

Thus, measuring the time spent looking at different aspects of a news post on social media allows us to investigate whether increased attention to certain elements factors into a users’ decision to further engage with the news post (e.g., through clicking or sharing). Similarly, a user may pay much attention to certain elements of a news post and decide not to engage with it further. Even in this case, attention to such elements may still be an explanatory mechanism for a lack of engagement.

In this study, we focus on characteristics of Facebook posts (i.e., comments) because user characteristics tend not to influence attention to Facebook news posts. Studies measuring news attention on Facebook find no correlation between users’ attention to news posts and their self-reported interest in certain news genres (Vraga et al., 2019) or political leanings (Sülflow et al., 2019). The latter study does find, however, that when the ideological slant of the first comment is in contrast with the leaning of the news post, one’s attention to the post increases. Furthermore, disagreement has been shown to pique curiosity and increase attention (Chen & Berger, 2013). Therefore, we hypothesize that:

Hypothesis 1: When users are presented with a post including a pair of comments in disagreement, their visual attention to the comments will increase.

Selection

Selection is the process of clicking a Facebook news post to read the full story. When selecting what content to read in a rich and varied content environment like Facebook, users partly rely on the impressions of other users, since ‘social arbitration is a widespread technique for dealing with the abundance of information in complex information sharing networks’ (Metzger et al., 2010, p. 433). Messing and Westwood (2014) find initial evidence for this social arbitration on Facebook. They argue that the presence of Facebook likes on a news story acts as a social endorsement that ‘trumps any effect’ relating to the article’s source or citizens’ partisan leanings (Messing & Westwood, 2014, p. 15).

However, Dvir-Gvirsman’s (2019) study finds that likes have no effect on news selection, which may be attributable to one of two factors. The first is time decay; Facebook likes may have lost their news selection effect as users have become accustomed to their presence on the platform. The second is that Messing and Westwood (2014) do not consider the role of disagreement in their study. An alternative explanation for their findings is that seeing social endorsements for a counter-attitudinal news post triggers a sense of disagreement, which in turn drives the selection of that news.

Historically, research suggests that citizens select news when disagreement is present in the form of conflict or controversy (Eilders, 2006). In addition, news articles high in controversy (Ziegele et al., 2018) have been shown to increase the commenting behavior of users in online environments. Given that comments seem to be an influential driver of news selection (Dvir-Gvirsman, 2019), and that disagreement provokes news selection, we expect that:

Hypothesis 2: When users are presented with posts including a pair of comments in disagreement, their likelihood to read the article associated with the post will increase.

Distribution

On Facebook, users can do more than just read the news. They can distribute news stories to their networks via the ‘share’ feature, and prior research suggests a marked difference between what users read and share online (Bright, 2016). Whereas the literature above suggests that controversy increases users’ likelihood to read a news story, several studies suggest that users tend to avoid sharing controversial news on social media. Trilling et al. (2017), for example, find that news items carrying a positive tone are twice more shareable on Facebook than negative-tone news. Similarly, Valenzuela et al. (2017) find that conflict frames in news items decrease sharing on both Facebook and Twitter.

One potential explanation for this trend is status-driven behavior, where the motivation for sharing certain types of content is to project a certain image, to maintain social relationships, and ultimately to increase one’s social status (Scholz et al., 2017). It follows that users would like to avoid sharing news surrounded by disagreement, out of fear of opening themselves to criticism from members of their social networks. We therefore expect that:

Hypothesis 3: When users are presented with a post including a pair of comments in disagreement, their likelihood to share the post on Facebook will decrease.

Hard versus soft news

Further supporting Hypothesis 3, Bright (2016) finds evidence of a ‘social news gap’ where news of more controversial nature (including politics, disasters, and crime) are less shared, but more read, than news on social welfare, science, and technology. In this social news gap, controversial news tends to drive reading and sharing behavior in line with the research outlined above; however, it also causes us to question whether the findings of previous research are biased towards news topics with societal ‘gravitas’: marijuana legislation (Winter et al., 2015), elections (Pierce et al., 2017), media coverage about suicide (Winter, 2018), and the refugee crisis in Germany (Sülflow et al., 2019).

These topics would traditionally fall under the genre of ‘hard news,’ which we understand as providing citizens with information about public affairs. ‘Soft news,’ meanwhile, refers to more leisurely pursuits such as entertainment, culture, sports, and weather. When studies introduce a distinction between hard and soft news, news engagement patterns exhibit more nuance. For example, Kalsnes and Larsson (2018) find that on Facebook in Norway, soft news posts (social issues, science, and crime) are more shared than hard news (stories on politics and economy). Therefore, we expect that disagreement in comments will affect attention, selection, and distribution differently across hard and soft news topics. However, there is an ongoing debate about what constitutes hard and soft news and which criteria should be followed when drawing the separation line between the two categories (Reinemann et al., 2016). This disagreement leads to an inconsistent application of the hard vs soft news distinction, which makes the comparison difficult across the various research pieces published using these concepts. Since the literature allows us to expect a difference but does not enable us to specify the direction of the relationship, we formulate the research question:

RQ: Will disagreement in comments affect user engagement with hard and soft news topics differently, and if so in which direction?

Method

To test these hypotheses and answer our research question, we chose a mixed-method approach combining a selective exposure experiment, eye tracking, and a post-test survey. We incorporate eye tracking to account for the inaccuracy of self-reported information in assessing the effects of information exposure. Unlike self-reported measures such as surveys (which provide coarse information that may be biased due to social desirability or memory fallibility), eye tracking measures provide a direct, objective, and precise observation of individual behavior that reflects how the process of attentional deployment unfolds. This includes both the components of cognitive processing that are conscious, as well as unconscious processes that significantly affect the behavior of respondents but are cognitively impenetrable (Holmqvist et al., 2011; Sülflow et al., 2019; Vraga et al., 2016).

Participants

A convenience sample of 96 participants (55% female), aged between 21 and 44 (with a mean of 24) participated in the study. A majority were students at Lund University (69% undergraduates, 24% master’s students). Participants were compensated with a café voucher. Results from a post-test survey revealed that participants considered themselves ‘very well’ or ‘well’ informed about current events (56%), or at least adequately informed (35%). The distribution of interests showed a clear preference for news about society and politics (e.g., 35% are very interested and 41% moderately interested in politics), paired with a lack of interest in sports (49% completely uninterested) and a moderate affinity for entertainment news (28% rather interested). In terms of their social media activity, 95% reported actively using Facebook, with 57% checking Facebook several times a day. These users reported engaging with content on the platform rather superficially: 55% reported having reacted to posts in the past week, but only 29% said that they commented and 11% shared a post.

Stimuli and apparatus

Twenty posts were selected from the Facebook page of the Swedish tabloid Aftonbladet, which we consider an appropriate case for the following reasons. Facebook is the most popular social media platform in Sweden: in 2018, 73% of Swedes used the platform actively, with 36% getting their news from the platform (Reuters, 2018). Aftonbladet is the largest news publisher in the country and the top-ranking source for online news, with approximately half of the Swedish population accessing the outlet at least weekly (Reuters, 2018). Aftonbladet is, moreover, the primary news source for 25% of Swedes aged 18–29 (Pew Research, 2018), which is the age demographic most closely aligned with our participants. In addition, because it is a tabloid, the presence of disagreement in Aftonbladet’s comment fields is realistic.

We gathered posts published by Aftonbladet during 2017 using the rFacebook package (Barbera et al., 2017) for the programing software R. The Facebook’s Graph API query returned 539 posts, which may be a subset of all posts made by the newspaper that year due to data limitations imposed by Facebook. From this dataset, we selected 20 posts with share counts close to the median of 49 (between a maximum of 82 and minimum of 23 shares). Selecting posts close to the median kept the level of newsworthiness relatively constant across posts.

Four posts were selected from each of five news genres: Entertainment, Economy, Politics, Society, and Sports. Entertainment includes stories relating to music, television series, and festivals. Economy covers topics relating to money, business, and banking. Politics relates to domestic politics and terrorism. Society refers to issues around health, crime, and education. Lastly, the Sports category captures stories involving famous athletes or sporting events. These topic categories guided our thematic selection of news posts and covered a comprehensive range of issues that a national tabloid would report as news.

To create the stimuli, we selected pairs of real Facebook comments that expressed similar (agreement) or differing opinions (disagreement) with one another – not the news post. Aware that the content of the news posts may influence the quality of comments (Stroud et al., 2015), we were careful to maintain an even tone in the comment field, regardless of the topic of the Facebook post. We then placed the comment pairs underneath the news post, with the second comment indented under the first (to signal a reply). The identity of the commenters was anonymized by altering the names and pictures of their profiles. No manipulation check was performed, since we consider the agreement and disagreement conditions clearly distinguishable.²

The stimuli were displayed on a 23.8’ EIZO FlexScan EV2451 monitor, while eye-movement data were recorded using a Tobii Pro Spectrum at 600 Hz. The experiment was conducted using a custom Matlab program making use of the PsychToolbox (Brainard, 1997) for visual presentation and the Titta toolbox (Niehorster et al., 2020) for control of the eye tracker.

Procedure

After a brief welcome, participants read and signed an informed consent form. Afterwards, the eye tracker was calibrated (average offset .78 degrees; no participants were excluded due to insufficient data quality), and participants were presented with written instructions about the experiment. Following a practice trial, which introduced participants to the experiment’s procedure (and is excluded from the analysis), each participant was shown the 20 individual Facebook posts. The topics of these posts were evenly distributed across the five news genres. Of the four posts within each genre, two contained the agreement condition and two contained the disagreement condition.³ The 20 stimuli can therefore be broken down as follows: (2 posts  × 2 conditions) × 5 genres = 20 stimuli. The order in which the stimuli were shown was randomized for each participant.

Participants could look at each post as long as they wanted. After each stimulus, respondents rated their likelihood to read and share the post on two continuous Likert scales (1–10). After viewing all 20 posts, participants were presented with an online survey asking about their level of political information, their news genre preferences, Facebook habits, and the importance they gave to comments when deciding what to read and share on Facebook.⁴

To determine where on the Facebook posts participants looked, fixation classification was performed offline on the recorded eye-tracking data using the I2MC algorithm version 2.0.0 (Hessels et al., 2017). Most parameters were left at their defaults, except that classified fixations shorter than 60 ms were removed, and that fixations closer together than 30 ms and 15 pixels were merged into a single fixation.

Each stimulus was divided into eight Areas of Interest (AOIs), with individual comments treated as two separate AOIs. Our main analysis includes only the 61% of trials in which both comments were inspected to ensure participants were exposed to the manipulation. Since Dvir-Gvirsman (2019) grouped all comments into a single AOI and Sülflow et al. (2019) included only a single comment in their study, we lack a benchmark to gauge whether this exclusion rate is within the normal gazing behavior for Facebook posts.

Gaze behavior was analyzed in the form of ‘dwells’; a dwell constitutes one or more consecutive fixations to the same AOI. Figure 1, below, shows the eight AOIs for the experiment stimuli.

Figure 1. Areas of Interest in a post from the entertainment category.

Since our stimuli were selected from actual Facebook comments, comment length varied across conditions and post genres. To account for this variation, the dwell time on comment AOIs was calculated as relative to the number of characters (i.e., standardized). We named this variable ‘relative dwell time on comments’, measured in milliseconds per character.

To categorize our stimuli along hard and soft news topics, we constructed a binary variable of post genres, contrasting hard news categories (Politics and Economy) with soft news categories (Entertainment, Society, and Sports). We checked the internal consistency of the soft and hard news categories through analyses of variance of each dependent variable and post genre. Although all three ANOVAs proved significant, pairwise post hoc comparisons revealed no significant differences within the hard news category. In the soft news category, significant differences were found in dwell time (Society vs Culture, p < 0.05), and in likelihood to read (Society vs Culture, p < 0.05 and Sport vs Society, p < 0.05). We attribute these differences to two main factors. The first is our stimuli selection process, which was constrained to selecting real news articles around the median share count of posts issued by Aftonbladet during 2017. The second relates to the news reporting itself. Stories pertaining to culture or sports can spill over to issues relevant for society and vice versa. We encourage readers to view our stimuli to become fully acquainted with our experimental design.

We also included in our models other variables, collected via the survey at the end of the experiment: participants’ self-reported interest in each of the five news genres (scale 1–5), frequency of Facebook use (scale 1–5), frequency of content sharing (binary variable describing if participants had shared content over the last week or not), and the perceived importance of post comments for deciding to share posts (scale 1–5).

In order to explain the variance in these dependent measures, we chose a multiple linear regression approach to generate inferential statistics. Since eye movement measures are characterized by a high level of individual variation (Holmqvist et al., 2011), we specifically used a linear mixed effects (LME) approach to test our hypotheses using the lme4 package in R (Bates et al., 2015). An LME approach allows us to model the effects of multiple predictors on a dependent variable, using separate intercepts for each participant (Long, 2012). Using data from our post-experiment survey, we followed a standard model simplification procedure where non-significant predictor variables were removed from the models unless they were required for tests of our hypotheses. Apart from investigating the fixed effects of predictors on the dependent variables (i.e., attention, selection, and distribution), the models also account for the random effects from individual participant and post genre.

Results

We first measured the dwell time and gaze sequence of each AOI and observed that the participants tended to explore posts from top to bottom. Irrespective of the comment condition, participants tended to look at the post, then at the comments, and only briefly (if at all) back to the post. Absolute dwell time on post (unadjusted to the length of the text) was about 8 to 10 sec, absolute dwell time on comments varied between 7 and 11 sec, and the dwell time on the post after having looked at the comments never reached two seconds. Thus, we did not observe any effect of the comments driving visual attention to the top half of the post, a result aligning with previous research (Dvir-Gvirsman, 2019).

We then ran LME models to test the strength of the relationship between disagreement and our three independent variables (attention, selection, and distribution). These models test Hypotheses 1–3 respectively, and their results are reported in the text in the corresponding sections below. To answer our research question, we performed additional LME analyses where we introduced the interaction effect of the hard–soft news distinction for each of the three independent variables. These results are reported in the following sections as tables. The complete results of all our models and code can be found in the supplementary material.

Attention

To test Hypothesis 1, we examined the effect of disagreement on visual attention to comments. Figure 2 illustrates the distribution of relative dwell time on comments by condition and topic, with the error bars denoting the standard error of the mean. An LME analysis revealed a main effect⁵ (p = 0.02) that relative dwell time was higher for comments in the disagreement condition, consistent with our expectation for Hypothesis 1.

Figure 2. Relative dwell time on comments by condition and genre.

To respond to our research question about possible differential effects relating to news genre, we introduce the hard/soft news distinction in a follow-up LME model. This model reveals a significant interaction (p < 0.01) between comment condition and post genre (Table 1). Further pairwise comparisons show that for soft news, dwell time on disagreeing comments was higher than for agreeing comments (t(1095) = 3.98, p < 0.01), whereas for hard news, dwell time to disagreeing comments was not different than to agreeing comments (t(1096) = 1.09 p = 0.27). Therefore, while post genre and post genre interest do not seem to increase visual attention to comments (p ≥ 0.1), we observe a significant difference in visual attention to comments between hard and soft news, and find that Hypothesis 1 only holds for soft news.

Table 1. LME analysis of relative dwell time on comments, interaction model.

	Estimate	Std. error	t-value	p(z)
(Intercept)	21.19	3.19	6.64	<0.01
Comment disagreement	4.51	1.13	3.99	<0.01
Post genre binary (hard news)	2.18	1.21	1.81	0.07
Interest in the post genre	0.42	0.35	1.19	0.24
Sharing driven by comments	3.49	1.25	2.80	0.005
Comment disagreement x hard news	−5.97	1.75	−3.40	0.001
$R_c^2:0.45$

In an attempt to further explain the possible causes of the direction of this interaction effect, we included all the trials in the dataset where participants looked at either one or two comments (N = 1624), to test if interest in the genre of the post covaried with the time spent on comments. An LME model showed that the more interested a respondent was in the news genre of the post, the higher the probability they looked at both comments (p = 0.006). Moreover, when we isolated visual attention to the content of the post only and tested with an LME model the effect of interest in the topic of the post on all 1920 trials, we found a small but significant positive effect (p = 0.02).

In sum, respondents’ genre interest had reliable positive effects on relative dwell time on comments and on the Facebook posts. Thus, these supplementary tests could not identify participants’ interest in post genre as the explanation for the interaction described in Table 1. The same table further shows that attention to the comment field is also driven by the self-reported importance of the comment field in participants’ decision to share content on Facebook (p = 0.005).

Selection

We operationalized selection as the self-reported intention to read the news article associated with each post. We asked participants to rate how likely they were to read the full news article associated with each stimulus (answers on a 0–10 Likert scale presented in Figure 3).

Figure 3. Answers to the read likelihood question.

Contrary to our expectation in Hypothesis 2, the likelihood to read the article was not higher in the disagreeing comment condition than in the agreeing condition (p = 0.43).

However, when examining our research question for selection, a significant interaction (p = 0.012) of post genre and comment disagreement revealed that comment disagreement depressed the likelihood to read the article significantly more for hard than for soft news (Table 2).

Table 2. LME analysis of likelihood to read, interaction model.

	Estimate	Std. error	t-value	p(z)
(Intercept)	1.88	0.32	5.79	<0.01
Comment disagreement	0.22	0.22	1.02	0.31
Post genre binary (hard news)	1.03	0.23	4.48	<0.01
Interest in the post genre	0.40	0.07	5.95	<0.01
Total dwell time on post	0.08	0.02	4.15	<0.01
Comment disagreement x hard news	−0.84	0.33	−2.50	0.01
$R_c^2:0.25$

Posts featuring content corresponding to participants’ news interests were associated with a strong positive effect on the likelihood to read the article associated with the post (p < 0.01). Moreover, if participants gave more visual attention to a post (total dwell time), they were more likely to read the article associated with it (p < 0.01). Across conditions, hard news topics were more likely to be read than soft news (p < 0.01). Taken together, these results show that individual preferences in news content and the amount of visual attention dedicated to Facebook posts contribute significantly to participants’ propensity to select a post for further reading.

Distribution

The distribution variable was defined as the self-reported likelihood to share a news item. Figure 4 shows that the overall likelihood-to-share ratings are low across experimental conditions and post genres.

Figure 4. Answers to the share likelihood question.

Nevertheless, the LME results indicate a significant main effect (p < 0.01), showing that stimuli in the disagreeing comment condition were associated with a decrease in likelihood to share the post, supporting Hypothesis 3.

We introduced a hard/soft distinction to answer our research question for distribution. Table 3 shows that users’ sharing behavior was not affected differently by hard vs soft news in the disagreement condition (the interaction test had a p = 0.15). In general, hard news were more likely to be shared than soft news (p = 0.02), and participants’ level of interest in the post genre also had a significantly positive effect on their likelihood to share (p = 0.03).

Table 3. LME analysis of the likelihood to share, interaction model.

	Estimate	Std. error	t-value	p(z)
(Intercept)	−0.21	0.82	−0.25	0.80
Comment disagreement	−0.28	0.14	−2.06	0.04
Post genre binary (hard news)	0.46	0.14	3.20	<0.01
Interest in the post genre	0.09	0.04	2.23	0.03
Relative dwell time on comments	0.003	0.003	0.76	0.45
Frequency of Facebook use	−0.28	0.14	−2.04	0.04
Frequency of sharing (binary)	1.23	0.57	2.18	0.03
Sharing driven by comments	0.57	0.15	3.81	<0.01
Comment disagreement x hard news	−0.30	0.21	−1.44	0.15
$R_c^2:0.49$

Among the self-reported variables on social media use, participants’ frequency of sharing (expressed as a binary variable encoding whether a participant had shared content on Facebook during the last week or not), the frequency of Facebook use, and participants’ perceived importance of post comments for determining their sharing decisions had significant effects on the likelihood to share posts (p ≤ 0.04).

Discussion and conclusion

Journalistic frames that promote negativity and conflict have been shown to influence news engagement on social media (Bright, 2016; Trilling et al., 2017; Valenzuela et al., 2017). We therefore sought to test whether disagreement between comments generates similar effects on how citizens view, read, and distribute posts on Facebook. Indeed, we show that comment disagreement can cue news engagement on social media, but it variably affects different news engagement behaviors. While comment disagreement increased visual attention to the comment field, it reduced sharing, and left unchanged the likelihood of a news article being read. We interpret these findings, firstly, to demonstrate the utility of segmenting news engagement into distinct behaviors.

Taking each behavior in turn, we find that comment disagreement increased visual attention to comments. This aligns with previous research showing that disagreement between the slant of a post and its comments generated higher dwell time on the post (Sülflow et al., 2019), and provides further evidence that comment fields can trigger cognitive involvement with user-generated debates online (Ziegele et al., 2018). However, comment disagreement did not cue participants to spend more time evaluating the main body of the post (i.e., the publisher’s text, image, and headline). We explain this primarily by the habitual way users peruse content on social media: in a top-down, linear fashion (Dvir-Gvirsman, 2019). Another explanation is users’ interest in the news genre: we found that the more interested a user is in the topic of the Facebook post, the more time that user spent gazing at that post. This finding contradicts Vraga et al. (2019), whose experiment found no relation between news interest and visual attention to posts. Differences in research design and national contexts may account for the lack of convergence in our respective findings.

Turning to our second news engagement behavior, news selection was not affected by comment disagreement. However, our inductive model simplification procedure revealed that interest in the post genre, and if the post was about a hard news topic, did increase the self-reported likelihood to read the news articles associated with the post.

For our third news engagement behavior, we found that comment disagreement decreased participants’ likelihood to share a post on Facebook. Overall, participants reported being unlikely to share any of our stimuli, but we still find that comment disagreement had a small, significant effect inhibiting users from sharing posts. We explain this tendency through both psychological and social factors. Chen and Berger (2013) find that the presence of discordant opinions increases a sense of psychological discomfort, which leads to conflict-avoidance. Moreover, sharing is a status-oriented behavior; people share news to strengthen their social bonds and to self-express in positive ways (Scholz et al., 2017). Sharing posts accompanied by disagreement may therefore be perceived as detrimental to users’ public image and to their amicable social relations with online ‘friends.’ If these findings are generalizable outside of our sample, trolling and flaming may indeed be effective methods to stunt the virality of important news on social media platforms built around fostering close-tie connections, like Facebook.

The answer to our research question about the difference between hard news and soft news topics is that, for two of the three news engagement behaviors (attention and selection), comment disagreement’s influence does vary by topic type. In comparison with soft news, our participants were less likely to devote visual attention to hard news comments when disagreement was present. The tendency of our respondents to look more at disagreeing comments to soft news can be explained by the same psychological mechanism uncovered by Chen and Berger (2013), who found that comments with dissenting opinions pique curiosity and interest. Since our participants had a strong interest in hard news, they appeared to be more curious in reading the community’s input about the soft news topics they knew less about. For news selection, we find that disagreeing comments reduced participants’ expressed intentions to read hard news over soft news articles. This fits with Bright’s (2016) claim that soft news is more likely to be read (but less shared) than news about politics. However, we did not find that the hard versus soft distinction mattered for sharing behavior, which was depressed overall by comment disagreement and seemed to be explained more by users’ interest in the news genre.

Methodologically, our study provides two contributions. The first is revealing the importance of comparing self-reported with observed measures. 70% of our participants reported that comments have little or no effect on their decision to share news on Facebook. However, our statistical models demonstrate that disagreement in comments did influence their visual attention and likelihood to share the stimuli. Although not explicitly instructed to do so, participants viewed both comments in 61% of trials and at least one comment in another 23%, signaling that users do take comment fields into consideration when evaluating news on Facebook. This discrepancy between participants’ self-reported and actual behavior concurs with the observations made by Haenschen (2019) and Vraga et al. (2016). Thus, we encourage researchers to complement respondents’ own perceptions with measures of their actual behaviors in future experimental designs.

Our second methodological contribution is creating stimuli from real-world Facebook posts and comments. The benefit of this approach is that it increases ecological validity by mimicking conversations that users might actually see on Facebook. However, the drawback is that it limited us from exaggerating the tone and degree of disagreement in the comments. Had we exacerbated the levels of disagreement in our stimuli, the effect sizes may have proved to be larger. Nevertheless, we encourage researchers to use data collected from platforms to avoid overstating the effects of what users actually encounter on social media.

Like other experiments that aim to recreate the Facebook environment, our study is limited in not being able to fully replicate the personalized user experience offered by the platform. In particular, our stimuli presented participants with only two comments, while they may encounter more on the platform itself. Still, the default setting for public pages (such as those of news organizations) is to show two comments underneath a post, and therefore our experiment approximates comments in cases of incidental exposure to news. In addition, real comment fields are likely to contain a mixture of agreement and disagreement. However, as the number of comments increase, so do the chances of disagreement between commentators. Thus, we consider our disagreement condition to approximate the majority of Facebook comment fields, and our findings to be extrapolated to the general conversation dynamic on the platform.

Our findings also have direct relevance for news outlets and other organizations, whose comments automatically appear as highly visible ‘top comments’ when they engage with their own posts. Since we find main effects of the disagreement condition on visual attention and sharing, organizations can reply to comments in agreement to facilitate sharing. News organizations could also benefit from correcting misinformation in the comments by replying and including a linked source, a particularly effective means of misinformation correction (Bode & Vraga, 2018). Not only would publishers defend the facts, but by inserting disagreement in the comment fields they may draw further visual attention from users. Even if organizations do not wish to comment on their own Facebook posts, our findings could help them better understand how agreeing or disagreeing exchanges between users affect attention and the sharing of their posts.

Disclosure statement

No potential conflict of interest was reported by the author(s). The authors gratefully acknowledge Lund University Humanities Lab, where the data were recorded.

Additional information

Notes on contributors

Anamaria Dutceac Segesten

Anamaria Dutceac Segesten is Associate Professor in Strategic Communication and Senior Lecturer in European Studies, Lund University.

Michael Bossetta

Michael Bossetta is Deputy Senior Lecturer in Media and Communication, Lund University.

Nils Holmberg

Nils Holmberg is Senior Lecturer in Strategic Communication, Lund University.

Diederick Niehorster

Diederick Niehorster is Researcher at the Humanities Lab, Lund University.

Notes

1 See the exact set-up for our Areas of Interest (AOIs) on page 15.

2 Our stimuli are published on FigShare at https://doi.org/10.6084/m9.figshare.11877363.v1.

3 Due to an error on our part, one post in the Sports category lacked a disagreement condition. All trials for this post (n = 49) are treated as agreement in the analysis.

4 The survey questions can be found on FigShare at https://doi.org/10.6084/m9.figshare.11877300.v1. The full dataset (eye tracking measures and survey results) is available at https://doi.org/10.6084/m9.figshare.11935623.v1.

5 For a full account of all statistical tests we ran, and their corresponding results please consult the R markdown file available at https://doi.org/10.6084/m9.figshare.11935677.v1.