Exploring the Relationship Between Stylistic Features and Reactions on Facebook: A Comparative Analysis of Newspaper Headlines and Status Messages

ABSTRACT

Social media platforms have become omnipresent over the last fifteen years and have forced news outlets to repackage the distribution of their content, custom-tailoring their news in accordance with the platform-specific social media logic. How news consumers engage with news outlets' content has also become more complex and more granular, as they can now express emotions that go beyond the “like-button” by “loving” or “caring” about specific (news) content. In this study, we link these Facebook Reactions to the specific features of headlines by conducting a content analysis of 3,163 news articles published by five Flemish news brands on Facebook. Results have revealed that status messages contain more emotional stylistic features as compared to Facebook headlines. Secondly, headlines and status messages comprising these emotional features were more likely to generate affective responses (as expressed in the emoji they use, “love”, “haha”, “wow”, “angry” and “sad”). In doing so, this study has examined Facebook Reactions’ functionality as a potential proxy of emotional engagement with specific headlines in the online news environment.

KEYWORDS:

• Content analysis
• emotionality
• Facebook reactions
• headline
• news sharing
• status messages

Introduction

In a highly saturated news environment, news outlets have struggled to attract attention and stimulate the readership of news consumers (de León and Trilling 2021; Holton and Lewis 2011). Attempting to increase contact with news consumers, news organizations have turned to social media platforms like Facebook, Twitter (now “X”), and Instagram to distribute their contents. As these platforms have become more institutionalized over the last fifteen years, they started to function as online hatches between news outlets' content and their news consumers (Poell, Nieborg, and Van Dijck 2019). This distribution on social media platforms has also resulted in the fact that many news outlets must compete for attention by creating engaging, shareable, and ready-to-consume content that appeals to readers' interests. This process of editing and “repackaging” content could potentially lead to the “softening of news” (Lamot 2022).

As a result of this repackaging process, the engagement and distribution of news content on social media platforms have become subject to academic attention. Work by, for example, Sturm Wilkerson, Riedl and Whipple (2021) has focused on “shares,” “likes,” and “comments,” revealing and hinting at the engagement of news content on social media platforms. Less attention, however, has been attributed to the Reactions feature of Facebook that was introduced in 2016. Through these reactions, news consumers can express in a more nuanced or granular way how they feel about certain content by clicking on one of the five icons: “Angry”, “Haha”, “Love”, “Sad”, and “Wow” – transcending the traditional “Like-button” (de León and Trilling 2021; Eberl et al. 2020; Tom, Pollmann, and Goudbeek 2018).

Emotions and news are intrinsically intertwined, as news consumers often rely on their friends and followers to filter out information, and “for what to make of the news” (Marquart, Ohme, and Möller 2020; Opgenhaffen 2021; Tom, Pollmann, and Goudbeek 2018). If a news story evokes a strong emotional response, people are more likely to share it with their network to express their feelings and connect with others who feel the same way (Berger and Milkman 2012). Additionally, emotion can help to create a sense of community and belonging among social media users. When people share news stories that align with their values and beliefs, they are able to connect with others who share those same values and beliefs (Martin and Nightingale 2019). This sharing and engaging with news content can foster a sense of community and belonging, which can be especially important for people who may feel isolated or disconnected in their offline lives (Lieberman and Schroeder 2020).

Wahl-Jorgensen (2020) and Lecheler (2020) have stated that over the past five years, there has been an “emotional turn” in journalism studies with increased emphasis on the relationship between emotionality and news content. However, Wahl-Jorgensen (2020) has added that there is a need to evaluate further how news consumers interact emotionally with the media, especially on sites like Facebook. In this study, we want to assess these Facebook Reactions as a potential proxy for emotional engagement in relation to specific headline features of news outlets. In addition, we want to understand how different news topics relate to specific emotional responses on Facebook. To make our case, we have focused on the five most prominent news outlets in the Dutch-speaking part of Belgium that were posted on Facebook between 13 January and 14 February 2020. The results of this study are valuable to advance our understanding of Facebook Reactions in relation to specific headline features of news outlets. For example, news eliciting sadness might have different engagement effects on audiences than news causing anger. Our findings reveal that almost all investigated features are significantly more common in status messages than in headlines (with the exception of mentions of locations and organizations). However, if we look at the engagement, we see that these features elicit more affective responses when used in the headlines compared to the status messages. In general, studying emotional engagement might help us to evaluate how audiences (dis)trust and perceive the media and its content. Additionally, it can shed light on how emotional responses to news content contribute to the formation of these perceptions.

Literature Review

Social Media Logic (SML) and the Distribution of News

To better understand the logic behind social media and its impact on news distribution, we draw on insights from the work of Van Dijck and Poell (2014), who introduced the concept of social media logic (SML). In this study, SML acts as an overarching theoretical framework to better identify the relationship between emotionality and the specific features of news headlines from the perspective of the news workers and the news consumers. Since emotions are conclusively linked to how users interact with content on social media platforms, it is important to briefly outline the four dimensions of SML and illustrate these to the news outlets' utilization of social media.

According to Van Dijck and Poell (2014), SML involves (1) programmability, (2) popularity, (3) connectivity, and (4) datafication. (1) Programmability refers to the ability of social media platforms to direct the production and consumption of news while also being influenced by audience behavior (see also, Altheide and Snow 1979, 1992). The logic of programmability of news outlets has become intertwined with how content is being distributed, where these outlets take both the algorithms, as well as the specific features of the platforms into account. (2) Popularity refers to the mechanisms that drive people or ideas on social media platforms. From the perspective of news outlets, their core functions as agenda-setters and gatekeepers are central to controlling and sustaining their popularity (see also, Aalberg, Strömbäck, and De Vreese 2012). (3) Connectivity refers to the algorithmic and automatic connections between individual users, accounts, posts, and advertisers. In the context of news outlets, connectivity on social media platforms must be linked to the direct contact between the content of these news organizations and the consumers of that same content. (4) Datafication refers to the ability to map and even predict users' preferences and behavior in real-time, which determines the selection and design of posts. For news organizations, datafication comes down to quantifying reading behavior by utilizing metrics systems (Lamot 2022).

Consequently, by applying the central principles of SML, news organizations can respond to the news preferences of the public by repackaging their articles according to the specific characteristics of various social media platforms, custom curtailing their content to its news consumers. Due to the emergence and the institutionalization of social media platforms, another central characteristic has emerged that can be linked to the four dimensions of SML in the light of news outlets and its news consumers, namely “interactivity” (Usher 2016). These platforms have made the interaction between news organizations and their news consumers less amorphous, resulting in a “direct feedback loop” in the form of user comments, likes, clicks, and shares (Lee and Tandoc 2017). News organizations are generally interested in ensuring that their posts are widely shared and potentially go viral on Facebook, as this allows these outlets to extend their reach beyond the regular group of loyal news consumers that engage with their content (Sturm Wilkinson, Riedl and Whipple 2021). Lamot, Kreutz, and Opgenhaffen (2022), for example, found differences between headlines on news websites and those selected for Facebook. The original headlines on news websites contained more persons, pronouns, and emotive words than Facebook headlines, contrary to assumptions about personalized and emotionalized social media. This evolution of more nuanced audience engagement has led to an abundance of studies that are focused on, but not limited to, audience metrics (Tandoc 2015) and the repackaging of news on social media (Opgenhaffen 2021; Vázquez-Herrero, Negreira-Rey, and López-García 2022).

Fewer studies, however, have focused on the SML of how audiences express their emotions in response to the content shared on social media platforms. Over the last five years, some studies have utilized audience responses to the material on social media platforms, including messages from political actors (Eberl et al. 2020; Jost, Maurer, and Hassler 2020) and news articles (Larsson 2018; Savolainen, Trilling, and Liotsiou 2020). Although the research of emotional engagement to content on social media platforms is still in a “preliminary stage”, several earlier studies in Communication Sciences have already hinted at the potential impact of emotions and news. In general, they have concluded that sensational news can generate different types of arousal and emotional responses (Grabe et al. 2000; Uribe and Gunter 2007) and that news invokes emotional responses in people concerning specific frames (Gross and D’ambrosio 2004). With the emergence of social media platforms, scholars have focused on the potential impact of these platforms on news distribution.

For our study, we believe that SML provides a comprehensive framework for understanding the role of emotionality on social media platforms in at least two ways. First, SML offers a way of understanding user engagement, and it helps us understand how emotions are strategically used to engage users. Emotions play a significant role in capturing users' attention, evoking reactions, and promoting interactions on social media (Jost, Maurer, and Hassler 2020). Second, SML helps us understand the exploration of emotional manipulation. SML offers therefore a valuable lens to investigate the strategies employed by social media editors to evoke specific emotions and shape user responses (Sturm Wilkerson, Riedl and Whipple 2021). As there has not been extensive research on SML in relation to emotionality, this study assesses headlines, status messages and their Reactions and whether they are a proxy for emotional engagement when it comes to specific characteristics of news headlines. In the next section of the literature review, the specific context and characteristics of Facebook and its Reactions function will be considered to underline the relevance of studying emotional engagement.

The Diversification of Facebook's “Like” Button

This study deliberately focuses on a platform where emotions can be measured more directly, namely Facebook and its relatively novel Reactions feature concerning the specific features of headlines. In 2016, the “Like” button feature was expanded in what was called “Facebook Reactions,” to include the following responses: “Angry”, “Haha”, “Like”, “Love”, “Sad”, and “Wow” (Stark 2019; Tom, Pollmann, and Goudbeek 2018). When the pandemic started in 2020, a seventh functionality was added, “Care”, represented as an emoji embracing a red heart (Sturm Wilkerson, Riedl and Whipple 2021). Facebook's expansion of its “Like” button can be interpreted in at least two ways. First, through these emoticons – or what now is known as “emojis” – specific feelings on Facebook can be expressed, which leads to even more granular or nuanced user data (de León and Trilling 2021; Larsson 2018; Tom, Pollmann, and Goudbeek 2018). Second, users on Facebook are obliged to only opt for one Reaction at a time instead of allowing more Reactions simultaneously (Masullo and Kim 2021). One of the main rationales for choosing one emotion is directly linked to a more “granular and targetable” profile that is utilized for selling advertisements.

Stark (2019) states that the diversification of the “Like” button made expressing emotions to online content central to the world's most dominant social media platform with almost 3 billion active users. The introduction of Facebook Reactions has inspired researchers to evaluate emotional responses concerning other more profound forms of engagement, like comments (Eberl et al. 2020). Over the last five years, some studies have increasingly focused on emotional responses to content on Facebook, including messages from political actors (Jost, Maurer, and Hassler 2020; Sturm Wilkerson, Riedl, and Whipple 2021) and news articles (Larsson 2018; Tom, Pollmann, and Goudbeek 2018). Jost, Maurer, and Hassler (2020) found that populist messages and negative representations of political actors generally increase the number of “Angry” Reactions. In contrast, more inclusive portrayals of political figures generally lead to more “Love” responses.

Similarly, the results of Sturm Wilkerson, Riedl and Whipple (2021) suggest that content on social media containing attacks and negative messages leads to more “Angry” Reactions from audiences on both left- and right-leaning pages on Facebook. Regarding news content, Larsson (2018) has found that negative news with relatively negative comments and Reactions is shared more than positive news with relatively positive responses. Tom, Pollmann, and Goudbeek (2018) conducted a sentiment analysis on news from The Wall Street Journal (N = 1.926), and they found that sentiment scores were systematically in line with the emoji that readers attached to the headline. However, they also found that positive headlines do not necessarily increase positive affective responses in comparison with neutral or negative news headlines.

The diversification of the “Like” button might imply that audience engagement is more valuable than a regular click, as a click might not (always) entail the actual interests of news consumers (see, for example, Groot Kormelink and Costera Meijer 2018). By utilizing Facebook Reactions, there might be a more valuable metric available to measure emotions of the audiences. In this regard, Sturm Wilkerson, Riedl and Whipple (2021) have labeled Facebook Reactions as affective affordances as they allow the expression of emotions in relation to content employing discrete, predetermined Reactions by the platform. Journalism scholars generally tend to work with popularity cues such as “Likes,” “Shares,” or “Comments.” However, due to the ambivalence of these features, researchers are unable to retrieve how audiences feel about specific news articles (Groot Kormelink and Costera Meijer 2018; Costera Meijer and Groot Kormelink 2019).

The introduction of the Facebook Reactions functionality might alleviate this issue, providing a possibility for researchers and news organizations alike to get a better sense of how users emotionally respond to news content. While some Reactions can be easily attributed to positive and negative emotions or effects such as “Love” (benevolent), “Angry,” and “Sad” (negative) (Kuo, Alvarado, and Chen 2018), this is not the case for all of them: for instance, “Haha” and “Wow” are ambiguous as they can either be positive or negative (e.g., ironic or sarcastic) (de León and Trilling 2021; Jost, Maurer, and Hassler 2020).

Therefore, it is worth delving into these types of Reactions concerning headlines in at least two ways. First, studying emotional responses, or what Sturm Wilkerson, Riedl and Whipple (2021) have called “affective affordances,” will advance our understanding of how audiences might “feel” about specific topics as they signal relevant emotions in the realm of journalism. Second, studying emotional responses to news content can help to evaluate why online polarization (Garimella and Weber 2017) and distrust in news outlets have increased over the last few years (Suiter and Fletcher 2020).

Drawing from these insights summarized above, some patterns in emotional engagement and specific headline features should be evident. For example, headlines containing negative sentiment will not be interpreted ambivalently by most news users and may arouse adverse emotional reactions such as “Sad” and “Angry.” On the other hand, headlines bearing a positive sentiment score will arouse more positive emotions, explicitly expressed through Reactions such as “Like” and “Love.” In the next section of the literature review, we reflect upon the relation between the repackaging of news content and emotionality. As news outlets are increasingly altering their content to the platform-specific rules of social media, it is valuable to take these adaptations into account.

News Content's Repackaging and Emotionality

In addition to the four central dimensions of social media logic (SML), news outlets also consider the “(re)packaging” of news, which can be seen as the fifth dimension of SML. This “repackaging” or adaptation takes place on different platforms—i.e., broader than just the social media platforms that news outlets are active on—and is defined as the process of alteration in which the design of the news content is changed (Erdal 2009; Lamot, Kreutz, and Opgenhaffen 2022 ). Earlier, Boczkowski (2005) described this process as recombining, recreating, and repurposing content that best suits the (social media) platform. Deuze (2006) called it bricolage, focusing on adapting and borrowing content to create something new.

The process of repackaging has become more institutionalized over the past decade as news companies realized that news consumers increasingly read news on social media platforms. Newsrooms have even created new job titles, such as social media editors, who are responsible for distributing and repackaging their content on the platforms that news outlets have accounts on (Cools, Van Gorp, and Opgenhaffen 2022; Lamot, Kreutz, and Opgenhaffen 2022). These social media editors adjust headlines and teasers on Facebook, Instagram, and Twitter to optimize the content based on the specifics of each social media platform (Lischka 2021). Due to the diversification of the “Like” button, these social media editors are now better able to observe the “emotional response” of their content as news consumers can engage with it in a more nuanced way. Research by Opgenhaffen (2021) has indicated that these editors are more inclined to attract the attention of news consumers by adding more subjective wording, for example, the headline or the teaser of the content. In doing so, the content becomes more of an “eye catcher” for the news consumer as it has been repackaged, potentially resulting in more engagement (Lamot, Kreutz, and Opgenhaffen 2022).

Kuiken et al. (2017) have underlined why studying headlines is necessary to understand online news content and emotional engagement on social media. They analyzed headlines from the newsletters of Blendle, an online newspaper kiosk from The Netherlands that clusters different news outlets' content, and compared them with the original headlines of these news outlets. They concluded that these headlines are reformulated and that they tend to be longer and include more emotionally charged words, quotes, and questions, pronouns, and signal words (see also, Opgenhaffen 2021). Although some of the headlines were only altered by one or two words, Kuiken et al. found that the repackaging of these headlines, like these pronouns and emotionally charged words, has a direct positive impact on audiences' click behavior. Lamot, Kreutz, and Opgenhaffen (2022) concluded that nearly half of the headlines on Facebook were adapted or (re)packaged from the original versions and that the rewritten headlines were generally simplified, with fewer characters and words but more exclamation marks. Similarly, Welbers et al. (2016) found that emotional engagement is one of the discerning features of the popularity of an online post.

The last-mentioned studies have shown that the presence of clickbait features in headlines has a direct influence on user engagement with news articles as these features are often conceptualized as exaggerated, sensational, and misleading (e.g., unresolved questions, dramatic phrasings). However, such studies remain narrow in focus dealing only with the relationship between repackaging and subsequent headline performance. Therefore, and in contrast with aggregated popularity cues, the data about Facebook Reactions usage allows for a more direct observation of the affective response of the reader toward these features. In this article, we track and evaluate how specific stylistic characteristics in headlines relate to the choice for a Facebook Reaction (Like, Love, Sad, Wow, Angry) on the part of the news user.

Detecting differences between status messages and headlines is crucial for understanding the nuanced dynamics of online communication and their implications for affective responses (Lamot, Kreutz, and Opgenhaffen 2022). While both convey information, status messages often afford greater space for personal expression and contextualization, potentially evoking stronger emotional reactions compared to the concise and often more neutral tone of headlines. This variance in emotional intensity may lead to differences in affective responses among audiences (Opgenhaffen 2021). Status messages, being more subjective and personalized, may elicit stronger emotional engagement due to their potential resonance with individual beliefs, values, and experiences. In contrast, headlines, designed for brevity and objectivity, may evoke more cognitive processing and less emotional engagement (a visualization of where the headline and the status message are located, see Appendix). By examining these differences between status messages and headlines, scholars can uncover the mechanisms underlying emotional engagement with online content and provide insights into how variations in communication styles contribute to polarization and the perception of media trustworthiness. Building on the studies of Kuiken et al. (2017) and Opgenhaffen (2021) on headline features, this brings us to the first research question and the following hypothesis:

RQ1: How do status messages and headlines on Facebook diverge on a range of stylistic features?

H1: Status messages contain more emotionality, operationalized as the use of (1) negative sentiment score, (2) more emotive words, (3) more emojis and (4) more exclamation marks than Facebook headlines

Furthermore, few studies have evaluated the role of these stylistic features and to what extent they arouse affective responses in terms of Facebook Reactions. As social media editors custom-curtail news, repackage their content, and share it on social media platforms like Facebook, we should also consider whether these stylistic features arouse specific responses in relation to topics. A topic like “war and disaster” might invoke negative emotional engagement like “Angry” or “Sad,” while a topic like “lifestyle” might trigger positive emotional responses like “Love” and “Wow.” Topics should therefore be considered as a relevant feature when evaluating emotional engagement, as they might be an important prerequisite for a Facebook Reaction. Our second aim is to explore the predictive power of various stylistic headline and status message features on the engagement and reactions elicited on Facebook:

RQ2: To what extent do these stylistic features influence affective responses, in terms of Facebook Reactions, with these status messages and headlines?

H2: Status messages and headlines containing these features will predict more affective responses, operationalized as the use of (1) more Love, (2) more Sad, (3) more Angry Reactions

Method

As illustrated, the focus of our study is assessing the difference in headlines and status messages in terms of stylistic features and the affective responses they are likely to generate. In the subsequent analysis, we address the Facebook posts of five influential Flemish news outlets beginning from January 13 until February 14, 2020. The posts under scrutiny derive from the broadsheet newspapers De Morgen and De Standaard, the popular newspapers Het Laatste Nieuws and Het Nieuwsblad, and the public service broadcaster VRT. We gathered data using CrowdTangle¹, a public insights tool owned and operated by Facebook that tracks content from publicly available profiles and pages. This resulted in a dataset containing 3,163 unique articles. Moreover, CrowdTangle allows for the collection of a specific set of metadata, such as which Page or public account something was posted from and how many interactions (e.g., likes, reactions, comments, shares) or video views it received (CrowdTangle Team 2020).

For analytical purposes, we seek to investigate whether certain stylistic features have an impact on affective responses toward social media headlines and status messages. Some analysts (e.g., Hagar, Diakopoulos, and DeWilde 2022; Lamot, Kreutz, and Opgenhaffen 2022; Tom, Pollmann, and Goudbeek 2018; Welbers and Opgenhaffen 2019) have attempted to draw fine distinctions between the writing style of websites and social media headlines by comparing them on features that include, but are not limited to sentiment, and average word length. In the same vein, we incorporate these features in our analysis as independent variables. We calculate the number of characters and number of words in the text units, and the average word length. Stanza,² a Python toolkit for Natural Language Processing (NLP), was used to automatically draw out the majority of stylistic features from the headlines. Specifically, we identify the presence of pronouns, references to persons, locations, and organizations. Similarly, we consider the presence of question words such as “wie” and “wat” (“Who” and “What” in English), and whether the text unit contains an exclamation mark or question mark. Furthermore, the Dutch-language LiLaH emotion lexicon by Ljubešić et al. (2020), which contains manual translations of 6,468 entries of the NRC Emotion Lexicon with their affiliated sentiment³ and emotion association⁴ (i.e., “disgust” → negative (sentiment), “anger” (emotion)), was used to map out the presence of certain emotive words by counting the absolute number of emotive words, and to calculate the average sentiment score by comparing the sentiment scores (negative or positive) of the present emotive words. Additionally, we use the lexicon to extract the predominant emotion of the text. Another efficient way in which people can express emotion on social media is through emoji. Therefore, we use the Python emoji library⁵ to count the number of emojis in the text. To account for varying lengths of the texts, we also calculate the relative frequency of emoji (number of emoji/ total number of tokens). Lastly, we employ Python scripts to calculate the occurrence of numbers (both digits such as 2, 3, and spelled out “two,” “three”) and quotes in the headlines (see also Lamot, Kreutz, and Opgenhaffen 2022). For an exhaustive overview of all features applied in the analysis, we kindly direct all readers of this manuscript to the appendix.

To provide further in-depth insights, the forthcoming analysis considers the topic of an article, which may yield a significant impact on the kind of Facebook Reactions the headlines are likely to receive. Drawing on the coding scheme administered by the Elektronisch Nieuwsarchief (ENA) in two decades of news content-analytical research, we analyze every article's topic according to 43 overarching codes, which we re-coded into twelve topic categories: (Inter)national Politics, Law Enforcement/Crime, Economy/Finance, Social Affairs, Wars/Disasters/accidents, Science/Technology/ Education, Mobility/infrastructure, Environment/Energy, Culture, Lifestyle/Travel, Media/ Entertainment, Celebrity. Next, we also classified the articles according to their genre. Coders could choose between eight genre categories being either (1) traditional (breaking) news items, (2) opinion pieces and editorials, (3) interviews, (4) feature stories/analyses, (5) video reports, (6) podcasts, (7) liveblogs or (8) other genres (listicles, slideshows, quizzes, …). The coding for these two variables was done manually by trained coders. Inter-coder reliability was calculated again on a random sample of 300 articles. While below the 10–15% threshold, Lombard, Snyder-Duch, and Bracken (2002) have argued that an adequate sample size for a reliability analysis will seldom need to exceed more than 300 units. Inter-coder reliability for this variable was assessed using Krippendorf's alpha, with satisfactory values above 0.70. Statistical analyses were performed using Stata software (version 17).

Results

In this section, we discuss to what extent stylistic features might lead to different patterns in emotional engagement with these headlines and status messages on Facebook. First, we will test whether we can find structural differences in the deployment of these stylistic features between Facebook headlines and status messages (Table 1). Next, we turn to regression results to see whether there are systematically different displays in affective responses related to specific features and whether these are independent of the topic an article is reporting on or the genre.

Table 1. Paired-samples t tests with Cohen's d.

	t	df	Cohen's d
Number of characters	−20.035***	3162	63.577
Number of words	−25.725***	3162	11.381
Average word length	24.736***	3162	1.824
Sentiment score	−9.651***	3162	.158
Number of emotion words	−8.922***	3162	2.146
Containing question words	−.640	3162	.278
Containing question marks	−3.312**	3162	.376
Containing exclamation marks	−9.255***	3162	.263
Containing quote	−6.884***	3162	.426
Containing number	4.337***	3162	.504
Containing pronoun	−18.848***	3162	.618
Containing person	7.954***	3162	.521
Containing organization	2.415*	3162	.456
Containing location	5.446***	3162	.488
Containing emojis	−11.327***	3162	.165

Note: A positive t-value implies that the mean is higher for the Facebook headlines. A negative t-value means that the mean of the Facebook status messages is higher.

*p < .05.

**p < .005.

***p < .001.

We first utilized a paired samples t-test to compare the headline and status message of the same newspaper articles. This decision was underpinned by our understanding of the inherent connection between the headline and status message as components of a single article, and our intention to accurately evaluate the impact of each on audience engagement in a second step. The results in Table 1 show how they significantly diverge on a range of stylistic features. First of all and not surprisingly, status messages tended to be longer than Facebook headlines, with the former containing a higher number of characters (M = 93.42, SD = 57.62), a higher number of words (M = 17.31, SD = 10.31), than the latter had in terms of characters (M = 70.77, SD = 24.12), words (M = 12.11, SD = 4.87); t(3162) =  −20.035, t(3162) =  −25.725 p < 0.001.

Of interest here are mostly the features that were said to contribute clearly to the emotionality of a particular status message or headline, such as the sentiment score, the number of emotive words and the relative frequency of emojis. We found that both status messages and headlines on average exhibited a more negative sentiment, albeit status messages were slightly less negative (M = −.023, SD = .108) than headlines (M = −.05, SD = .130). Yet, status messages on Facebook contained relatively more emotive words (M = 2.09, SD = 1.750) and emojis (M = .033, SD = .16) as compared to headline of a news article on Facebook (M = 1.75, SD = 1.349); t(3162) =  −8.922 and (M = .00017, SD = .003); t(3162) =  −11.327, p < 0.001. Emotionality may subsequently be reinforced by the injection of exclamation marks, which Welbers and Opgenhaffen (2019) claim to be conveying some sense of emotion as they allow the author to emphasize issues more vividly. Exclamation marks were more prevalent for Facebook status messages (M = 0.06, SD = 0.239) than for the headlines (M = 0.02, SD = 0.132); t(3162) = −9.255, p < 0.001.

Moreover, concerning the other stylistic features extracted from the headlines and status messages, we adopted a quite explorative approach since no specific hypotheses could be generated based on previous literature. Intuitively, we would expect that emotionality could manifest itself in the use of direct quotations and personalized storytelling (by adding references to persons, organizations or locations). From the table, it can be seen that status messages (M = .14, SD = .352) used more quotations than headlines (M = .09, SD = .290), t(3162) = −6.884, p < 0.001. Moreover, they also had more references to pronouns, t(3162) =  −18.848, p < 0.001. Contrary to what we expected, however, named entities such as reference to persons, organizations and locations were more prominently featured in the headlines (p < 0.05). Status messages contained on average fewer references to persons (M = .24, SD = .425), organizations (M = .14, SD = .352) and locations (M = .17, SD = .379) while headlines contained a significantly higher number of these references (M = .31, SD = 463; M = .16, SD = .371; M = .22; SD = .415). Lastly, numbers have been linked to clickbait headlines as well (Kuiken et al. 2017) and might add emphasis to the magnitude of an event. Headlines contained on average more of these numerical references than status messages.

We thus can confirm our H1 stating that status messages contain more emotionality as compared to headlines. They do contain more stylistic features that are unambiguously linked to emotionality (emotive words, exclamation marks, sentiment) (see also Opgenhaffen 2021). Headlines, on the other hand, contained more numbers and references to persons, organizations and locations.

The Relationship Between Features in Headlines or Status Messages and Facebook Reactions

The second research objective dealt with the question of whether the writing style of headlines and status messages has an impact on subsequent affective responses to headlines. Using negative binomial regression analysis, we tested the relationship between different stylistic features in headlines and status messages and the number of Facebook Reactions. The number of social media reactions typically follows a count distribution (see also Saxton and Waters 2014; Trilling, Tolochko, and Burscher 2017), with positive and highly skewed to the right, which is why we prefer negative binomial regression over, for example, Poisson regression. Table 2 presents an overview of all statistical associations. We will discuss the most prevalent outcomes in the section below.

Table 2. Negative binomial regression with Facebook Reactions as dependent variable (Model I)

	Like	Love	Haha	Wow	Sad	Angry
Headline_no_characters	1.001	1.001	.994	1.019***	1.032***	1.026***
Status_no_characters	.992**	.982***	1.003	.995	.996	1.003
Headline_no_words	.981	.951	1.061**	.898***	.838***	.875**
Status_no_words	1.026†	1.08**	.962†	.994	.984	.960
Headline_avg_wordlength	.994	1.102	1.013	.922**	.901**	.989
Status_avg_wordlength	.947	1.058	.768***	.999	.926	1.023
Headline_sentiment	1.826**	3.116***	3.18***	.633†	.341**	4.97***
Status_sentiment	1.876*	3.746***	1.364	.898	.425*	.175***
Headline_no_emotionwords	.991	.914***	1.041	1.15***	1.15**	1.371***
Status_no_emotionwords	1.012	1.021	1.003	1.02	1.10**	1.06
Headline_questionwords	.593***	.334***	.676	.754^†	.583*	.327***
Status_questionwords	.587***	.405***	.708*	.597**	.798	.335***
Headline_questionmarks	.692***	.661**	.639**	.996	.831	.551**
Status_questionmarks	1.16†	.962	.919	.575***	.358	1.01
Headline_exclamation	1.909**	1.39	1.88*	2.492***	.412*	2.007
Status_exclamation	2.92***	3.72***	1.22	1.498**	.953	.579
Headline_quote	1.286**	1.75***	1.360*	1.077**	1.15	1.012
Status_quote	1.186*	1.078	1.17	1.014	1.078	1.27†
Headline_number	1.375***	1.446**	.773**	2.369***	2.58***	1.75***
Status_number	1.002	.727**	.83^†	1.478***	1.795***	.935
Headline_pronoun	1.19**	1.604***	.893	1.137†	.869	1.004
Status_pronoun	1.054***	.984	1.136	1.125^†	1.479***	1.373**
Headline_person	1.406***	1.81***	1.59***	.975	1.306*	.720**
Status_person	.896	.840	.769**	.703***	.994	.686**
Headline_organization	.692***	.541***	.992	.684***	.238***	.977
Status_organization	.956	1.178	.994	.579***	.661**	.58***
Headline_location	.912	.687***	.805*	1.468***	1.61***	.872
Status_location	1.01	.905	.867	1.092	1.16	.456***
Constant	299.44***	36.61***	100.635***	54.713***	77.439***	30.937***
AIC	38,021.13	22,671.9	24,499.1	27,300.97	22,316.05	21,925.31

Note: Cell entries are incidence-rate ratios.

^† p < 0.1, *p < 0.05, **p < 0.01, ***p < 0.001.

First, we try to distinguish the “net effect” of our stylistic features on engagement. Overall, the following features seem to amount to the number of Facebook interactions that both a headline and status message are likely to generate: containing positive/negative sentiment, containing emotive words and containing exclamation marks. Rather intuitively, a positive association was found between headlines and status messages containing a positive sentiment score and more “positive” Facebook Reactions such as “Love,” “Like” (p < 0.05). For headlines, an increase in positive sentiment was associated with up to an 82% predicted increase in Likes, whereas for status messages, the predicted Likes surged by 87%. The effect was even larger when it came to Facebook Love Reactions, increasing more than twofold for status messages (274%) and headlines (211%) containing positive sentimental words. In contrast, negative responses such as “Sad” and “Angry” Reactions (p < 0.05) were associated with negative sentiment in both headlines and status messages. Similarly, headlines containing exclamation marks tend to have a significantly higher number of “Likes” and “Wows.” A 90% increase in Likes and a 149% in Facebook “Wows” was recorded for headlines containing an exclamation mark, whereas inserting exclamation marks in status messages occurred with successive increases of 193% in “Likes” and 150% in Facebook “Wows” as well.

It is worth noting that some features also seemed to perform reasonably well for headlines. Interestingly, integrating numbers, quotes and emotive words in headlines has a positive correlation with the number of both positive and negative affective responses (only for “Haha” there was no measurable effect) that a particular headline was likely to receive. The engagement with headlines that contain numbers is on average 37–158 percentage points higher than headlines without the feature (p < 0.01). Similarly, adding quotes did produce strikingly higher amounts of positive responses “Like,” “Love,” “Haha,” and the more ambivalent Facebook “Wow,” up to 75 percentage points. Furthermore, adding emotive words in headlines was significantly related to an increase in negative emotional responses “Sad,” “Wow,” and “Angry” Reactions of up to 15–37%. These findings thus lend empirical support to our second hypothesis stating that emotional appeals in headlines and status messages predict affective responses.

However, it is worth looking at more than stylistic features alone, especially since the complexity of news articles extends far beyond headlines and status messages, encompassing the topic, genre or other characteristic within the content itself. These elements collectively influence the reader's engagement and the subsequent reactions generated. We therefore incorporated a nuanced approach by including a second model in our regression analysis (in Appendix). In this model, we controlled for additional variables such as topics and genres of the news content, recognizing their potential influence on Facebook Reactions. We inserted dummy variables for both controls, where “Politics” served as the reference category for topics and the “inverted pyramid news item” for genre.

The findings from this second model are particularly illuminating. Despite accounting for the variability introduced by topics and genres, we observe that some, but not all, of the effects of the aforementioned features have at least some “net” effect on predicting Facebook Reactions. This outcome underscores a significant point: while the substance of a news article (being its topic and genre) undoubtedly influences engagement, the stylistic elements of headlines and status messages hold a distinct and powerful effect on Facebook Reactions, independent of content type. Specifically, some of our findings reveal a slight distinction in how “emotional features” (sentiment, emotive words, exclamation marks) and “informational features” such as named entities (i.e., persons, organizations and locations) predict user engagement across these two formats. For emotional features the effect on engagement often seems to be more pronounced for status messages, suggesting that the personal and direct nature of these messages amplifies the impact of affective responses in terms of Facebook Reactions. Conversely, the inclusion of “informational features” shows a somewhat stronger positive association with engagement for headlines.

Some of the control variables exert an influence on engagement as well. One of the most striking results to emerge is that while the associations between the different Facebook Reactions and the topic categories mostly tended to be positive, showing that these topics are, in general, more likely to enhance Reactions such as “Likes,” “Wows” as compared to the reference category, we see an opposite tendency for the Facebook “Haha” and “Angry” Reaction. Facebook “Haha” tended to decrease for all topic categories aside from political news and lifestyle news. This affective response is the opposite emotion that one would expect to see related to political news, yet, it might indicate irony or sarcasm, as mentioned earlier in this study (Kuo, Alvarado, and Chen 2018). Concerning genres, we find a negative association between almost all genres and Facebook Reactions, indicating that news users mostly react to the traditional, inverted pyramid news item compared to the other genres (interview, feature, video item, live blog, opinion piece, podcast, listicals).

Discussion

Prior studies have evaluated how Facebook status messages look in terms of stylistic features as compared to the headlines of the particular news item (Welbers and Opgenhaffen 2019). However, very little was found in the literature on the affective response these features tend to evoke for news users concerning topics and genres. The purpose of this paper was to fill this gap in the literature by providing empirical evidence on how headlines and status messages affect news users' emotional responses. Specifically, we have examined the potential of the Facebook Reactions functionality as a proxy of emotional engagement with specific headlines in the online news environment.

As a first step in our empirical analysis, we addressed the question of whether Facebook status messages contain more emotional elements compared to Facebook headlines of news articles. Specifically, we compared the number of emotive words and sentiment (i.e., positive vs. negative) of status messages to the classic headlines, using the Dutch LiLaH emotion lexicon, and compared the relative frequency of emoji in the texts. Our results reveal that the status messages in Facebook posts of newspapers contain more emotive words, more emojis, and a slightly more positive sentiment (albeit the tone on average is mostly negative) than the headlines, thus supporting our hypothesis.. This echoes the findings of Welbers and Opgenhaffen (2019) who showed how newspapers use status messages to add subjective expression to news on social media. For future research, we suggest not only looking at the number of emojis present in the text, but also using sophisticated lexicons to map the sentiment of these emojis (e.g., positive, negative, neutral) to the engagement with the post. To the best of our knowledge, there currently is no universally applicable, up-to-date lexicon for emoji sentiment categorization. The dynamic nature of emoji addition and usage, coupled with the diverse interpretations across different platforms, time periods (for instance, the skull emoji () was traditionally associated with danger, death, or negativity, but is now increasingly used to convey humor or sarcasm in light-hearted context), and demographic groups, suggests that any attempt to standardize emoji sentiment must be flexible, regularly updated, and sensitive to the nuances of digital communication culture.

The second question in this study sought to determine to what extent stylistic features impact the engagement a headline or status message is likely to generate. This investigation stems from previous empirical work that reported how stylistic features directly impact engagement with news headlines (Hagar, Diakopoulos, and DeWilde 2022; Kuiken et al. 2017; Lamot, Kreutz, and Opgenhaffen 2022). Comparison of the findings with those other studies confirms how the presence of punctuation marks, named entities, the inclusion of numbers, and emotional words are all characteristics of a “popular” headline or status message. The yields in this investigation were higher than those of other studies since it attempted to adopt a more nuanced approach by distinguishing between the different Reactions to overcome the limitation that most research relies on vague, aggregated popularity cues as a proxy for audience interest. It hereby builds on the work of Tom, Pollmann, and Goudbeek (2018) that attempted to evaluate the emotional response of readers to affective headlines. Just like their study, we systematically related Facebook Reactions to specific affective features in headlines, yet, we also included status messages in our analysis. By analyzing both these attention-luring formats in unison and broadening the number of stylistic features incorporated in the analysis, this study therefore offers a more complete picture of how news is presented and consumed on social media, capturing the nuances of how different elements interact to affect user engagement.

We held no firm hypotheses concerning the other features due to the lack of direction from theory or past empirical work, yet we found some novel results. Headlines and status messages with a positive sentiment are related to an increase in positive emotional responses by news users (as expressed in the emoji they use, “Like,” “Love” and “Haha”) and a reduction in negative affective responses (“'Sad”). Adding emotion words on the other hand, evokes an increase of negative affective responses (e.g., negative emoji such as “Sad,” “Angry”) primarily when they are inserted in the headlines. This largely mirrors what Tom, Pollmann, and Goudbeek (2018) illustrated as well.

Additionally, the use of exclamation marks had a positive association with positive affective responses (“Like,” “Love,” “Wow”). A possible explanation for these results may be that exclamation marks used in online communications usually have a more positive emotional intensity (Hancock, Landrigan, and Silver 2007). Overall, exclamation marks are used to increase the strength of an expression (Teh et al. 2015), which is why it is also plausible that they would reinforce strong negative emotions such as “Angry.” Sturm Wilkerson, Riedl and Whipple (2021) similarly found that messages containing exclamations have a significant relationship with ambivalent emotions such as “Like,” “Haha,” and “Wow.” Their study proved to be a valuable entry point for us, since it was one of the first attempts to study the relationship between stylistic features and emotional appeals on Facebook. However, the study focused on hyperpartisan US media outlets producing political news in the 2016 election period, making it a most likely case to find strong emotional appeals. Extending their analysis to legacy news media outlets in Belgium's less polarized context enriches the research by providing a more varied and potentially more representative exploration of how stylistic devices influence emotional appeals on Facebook.

Of course, focusing solely on these stylistic features disregards the fact that the user's emotional response to a news item depends on more than just exposure to a headline or a status message. Understanding an article's context is equally vital besides its particular writing approach. To better understand the relationship with engagement, we also evaluated topics and genres in relation to Facebook Reactions. The results in this study also accord with earlier observations by Larsson (2018), which showed that “softer” topic categories dealing with Lifestyle, Media and Entertainment, Environment (which on Facebook primarily deals with animals in this sample) induce much more positive user reactions such as “Love” and “Wow.” Politics, by contrast, aroused more “Haha” and “Angry” Reactions. Jost, Maurer and Hassler (2020) and Kuo, Alvarado, and Chen (2018) contended that the sentiment of “Haha” could also be ironic or sarcastic, while “Angry” displays an indisputable negative sentiment. The fact that political news in the period under study was primarily associated with rather negative or ambivalent emotional reactions, might signal some sort of dissatisfaction with politics.

The robustness of the features in this model indicates, however, that the stylistic features in headlines and status messages have a compelling impact on capturing user attention and eliciting reactions, even when controlling for topic and genre. It suggests that the manner in which information is presented to readers—through specific stylistic choices in headlines and status messages—can significantly influence the likelihood of initial engagement, regardless of the article's topic or genre. Additionally, our findings highlight the dynamics at play between different stylistic features and the format (headline or status message) through which they are conveyed. The differential impact of more “emotional features” (such as exclamation marks, sentiment and emotive words) (see also Opgenhaffen 2021) and “informational features” (such as named entities “organizations, locations, persons” and quotes) across status messages and headlines suggest a strategic complementarity, where the effective use of stylistic features should be tailored to the communication format to maximize Facebook engagement.

Conclusion

The introduction of the Facebook Reactions functionality has transformed social interaction with news content, providing users with more nuanced means to express their emotional appeals to news content beyond the simple “Like,” including diverse reactions such as love, amusement, sadness, anger, and surprise. We chose to study these reactions in relation to headlines and status messages, as an increasing amount of news consumers tend to react to the content of the platform, rather than “clicking-through” and reading the content on the news website itself. The study currently encompasses a dataset spanning one month with more than 3000 data points, providing a solid case study for initial insights. However, to uncover potential trends in affective responses to news content, it would be beneficial to extend the scope of data collection by scraping Facebook posts over a longer period. This expanded dataset could reveal more nuanced patterns and shifts in public sentiment, offering a deeper understanding of the dynamics at play.

Because these Facebook Reactions might potentially serve as an emotional shorthand, assisting news users in expressing their feelings and attitudes during news consumption, this study contributes to the earlier studies of the “emotional turn” in journalism (see, for example, Wahl-Jorgensen 2020) and paves the way for a more normative discussion about the feelings news organizations want to provoke with their content. It can be of interest to journalism scholars and news media organizations to better understand the relationship between engagement and the distribution and moderation of news in a more nuanced way. The results suggest that insights from Natural Language Processing (NLP) can be of interest to news organizations, helping them understand how audiences engage with their content and helping them to craft their headlines and status messages in more nuanced ways (see also Hagar, Diakopoulos, and DeWilde 2022). However, we would like to refrain from overemphasizing the stylistic aspect to avoid practitioners generalizing them as the alpha and omega of news packaging. Involving and engaging your audience begs for more than clinging to certain stylistic features in headline writing and status message crafting.

However, one important conclusion to be drawn from this research is that the interpretation of Facebook Reactions as clear indicators of emotional appeal comes with its limitations and requires careful consideration of inherent ambiguities. The primary constraint lies in the ambivalence of these reactions themselves. While Facebook Reactions may indicate some affective response toward news, they are still too crude to account for the variance in individual attitudes to the Emoji meanings of the platform. The unexpectedness of some of our results (e.g., the Facebook “Haha” emoji in relation to political news) underscores the complexity of fully grasping emotional responses of news users. For instance, even less ambiguous emotions such a “Sad” reaction can signify empathy with the protagonists of a news story, personal sorrow related to the news topic, or even disappointment with the journalistic quality or framing of the article. Similarly, an “Angry” reaction might reflect irritation at the situation or persons reported about (e.g., the Angry Reaction in relation to political news), frustration with the news organization, or even disagreement with other users' reactions. This ambivalence underscores the complexity of deducing precise emotional states or stances from Facebook Reactions, as they encapsulate a wide range of individual interpretations and motivations (see also Smoliarova, Gromova, and Pavlushkina 2018). Future work might reduce this ambivalence in affective reactions for instance, by combining an analysis of the emoji's with a content analysis of comment sections to detect what the anger is directed toward. More broadly, a qualitative user-centered approach is also needed to establish a better understanding of how emoji are interpreted and deployed by news users in their interactions on social media.

Furthermore, another significant contingency in relying solely on Facebook Reactions to gauge audience sentiment is the potential oversight of a substantial segment of the news audience. First of all, the Like button is still the default option. It likely causes certain demographics less familiar with Facebook to go for “Like” more often than other responses. Second, researchers have argued how on social media platforms such as Facebook there often is a discrepancy between a “vocal minority,” who is eager to engage on the platform in terms of comments, shares and likes and a “silent majority” of “engaged listeners” (Barnes 2014; Beckers et al. 2021) who, despite perhaps experiencing strong emotional responses to news content, may choose not to express these feelings through Facebook Reactions. The differential emotional appeals between these two groups can significantly affect the interpretation of Facebook Reactions as a proxy for public sentiment toward news. The vocal minority, by virtue of being more likely to express their emotions publicly, may skew the perceived emotional reception of a news article. Beckers et al. (2021) for example, found how journalists tend to have a right-wing bias in their perceptions of public opinion. When asked why their perceptions are biased toward the right, the journalists almost consistently referred to the audience reactions they received on social media. Consequently, if journalists systematically misperceive the emotions of news users, this may have unwanted effects on their reporting and perception of certain issues.

In conclusion, while Facebook Reactions offer a useful lens to study the emotional engagement of users with news content, they also present challenges in terms of ambivalence and representativeness. Recognizing these limitations is crucial for researchers and media professionals aiming to understand the complex dynamics of audience engagement with news on social media. A comprehensive analysis of news content's emotional impact should, therefore, consider not only the explicit reactions of users but also account for the subtleties of reaction ambivalence and the silent voices in the audience landscape.

Disclosure Statement

No potential conflict of interest was reported by the author(s).

Notes

1 https://www.crowdtangle.com/

2 https://stanfordnlp.github.io/stanza/

3 Binary: positive or negative.

4 Anger, anticipation, disgust, fear, joy, sadness, surprise, and trust.

5 https://pypi.org/project/emoji/