Fact-Checking and Audience Engagement: A Study of Content Analysis and Audience Behavioral Data of Fact-Checking Coverage from News Media

Abstract

This study examined (a) what message variations characterize news articles that fact-check (mis)information (N = 914) and (b) how those message features shape audience engagement with the articles. The study content-analyzed fact-checking coverage from major news outlets in South Korea using both manual and computerized coding, focusing on three categories of message characteristics: source transparency, contextual information, and vividness. The content-analysis data were examined in relation to behavioral data of audience engagement (“like” and “angry” reactions, shares, and comments) on Naver News, the most popular news portal in South Korea. Using statistics and official reports as evidence and specifying when the claim at hand was made facilitated audience engagement behaviors. News articles triggered more audience comments when they (a) mentioned the importance of fact-checking the claim under scrutiny, and (b) conveyed negative content. Findings are discussed in light of the empirical and practical implications of the current efforts to fight “fake news” by news media.

Keywords:

• Fact-checking
• audience engagement
• source transparency
• contextual information
• vividness
• journalism
• message effects

Fact-checking is “an emerging genre of journalism,” which reports “the accuracy of a claim or a text that is already in circulation” (Graves and Amazeen 2019). Although fact-checking has become a popular journalistic practice across the globe over the past decade (Graves 2018), previous research has largely focused on the motivations, purposes, or principles of fact-checking (Graves, Nyhan, and Reifler 2016), while another line of research has examined the effectiveness of fact-checking in correcting misinformation (Walter et al. 2020). Despite these scholarly endeavors, virtually no research has conducted a systematic content-analysis of fact-checking (cf. Humprecht 2020), resulting in a dearth of knowledge on what content features characterize this new journalistic practice. As Singer (2021) emphasized, to enhance our understanding of fact-checking journalism, an empirical investigation of “what they actually produce and not just what they describe themselves as producing” (16) is warranted.

Moreover, little is known about message factors that affect news audiences’ engagement with fact-checking coverage, such as “liking,” commenting on, and sharing the content. Audience engagement with fact-checking coverage is an important area of investigation, because news consumers routinely observe and are strongly influenced by fellow consumers’ engagement behaviors online. Audience engagement is an essential part of the fact-checking process also because, as with other news genres, it is ultimately through audience engagement that fact-checking can be widely accepted and disseminated in the current media environment.

This study is an exploratory investigation of the message characteristics of news articles that fact-check (mis)information and their relationship with audience engagement. We content-analyzed fact-checking coverage from major news outlets in South Korea (N = 914 articles), with a focus on (a) message features that are known to be important ingredients of high-quality news in general and of fact-checking coverage in particular, and (b) those diagnostic of persuasive and viral effectiveness. Next, we examined how frequently these message features were employed in fact-checking coverage, and then integrated the content analysis data with behavioral data of audience engagement such as “likes,” comments, and shares on Naver News, the most popular online news portal in South Korea (Newman et al. 2020). Lastly, we tested how the message characteristics of fact-checking coverage affected audience news engagement behaviors.

Characteristics of Fact-Checking Coverage

News articles that report fact-checking “directly evaluate the accuracy of substantive claims” (Graves, Nyhan, and Reifler 2016, 102) and thus serve as interventions to reduce or inoculate individuals from the influence of misinformation (Amazeen 2020). Fact-checking in this study refers to news organizations’ efforts to verify (mis)information that is already in circulation in the public communication environment (i.e., external fact-checking; Graves and Amazeen 2019). As journalistic fact-checking spread across the globe, it came to cover a broad range of issues, from statements by politicians, to internet rumors. While news outlets’ fact-checking coverage shares many message characteristics with other news genres (e.g., exemplification; Zillmann and Brosius 2000), it also includes some unique components, such as the description of the claim under examination, the verdict of the evaluation, and the process and evidence used to reach the conclusion (Humprecht 2020).

We focused on three categories of message features of fact-checking coverage that are (a) prominent in this emerging genre of news and (b) expected to enhance its effectiveness in verifying (mis)information. Source transparency and contextual information are two categories that have been emphasized by research on journalistic fact-checking (e.g., Graves 2017; Humprecht 2020), while vividness has been underscored by research on persuasive and viral message effectiveness (e.g., Berger 2014; O'Keefe 2002).

Source Transparency

Transparency is the degree of openness about the news production process (Humprecht 2020) and it is considered an important predictor of news performance and journalistic professionalism (Humprecht and Esser 2018). We specifically focus on the transparency of information sources used by journalists to judge the veracity of a claim at hand. Within the context of fact-checking, as Graves (2017, 528) put it, “working transparently requires that all sources used to analyze a claim be revealed to the reader.” Source transparency refers to the extent to which fact-checkers explicitly provide evidential sources (e.g., statistics) that they referenced in their verification processes, helping news consumers understand on what basis they made the judgment (Humprecht 2020). A transparent approach of showing the audience the process of claim verification is consistent with the fundamental idea of fact-checking and contributes to the objectivity of the journalistic practice (Graves 2017; Singer 2021). In the current study, we examined how frequently fact-checking coverage explicitly mentioned the following evidential sources to support its verdict(s): statistics, official reports, laws/rules, news reports, press releases, websites, and interviews (Humprecht 2020).

Contextual Information

The emphasis on contextual, or background, information lies at the heart of fact-checking journalism. Fact-checking can be viewed as an attempt to overcome the limitations of the mechanical objectivity norm (Graves 2016), because journalistic fact-checking focuses on the “why” above and beyond the “who-what-when-where” of descriptive journalism and provides interpretive and contextual information (Salgado and Strömbäck 2012). Additionally, illustrating why the claim is worthy of fact-checking can also convince critics of fact-checking by justifying the choices of claims being verified (Graves 2017). We focused on two types of contextual information regarding the claim under evaluation, an essential component of fact-checking coverage. First, we ascertained whether the fact-checking coverage discussed why the claim under evaluation occurred. Second, we examined whether the coverage explained why fact-checking the claim was important (Salgado and Strömbäck 2012).

Vividness

The third group of content characteristics we investigated was centered on the notion of vividness (Nisbett and Ross 1980), which has been studied as a stylistic feature of news reports that shapes audiences’ perceptions and judgements about reality (Zillmann and Brosius 2000). The vividness hypothesis posits that vivid messages are more likely to be encoded, accessible, and recalled in recipients’ memory, hence influence subsequent judgment to a greater degree than their less vivid counterparts (Nisbett and Ross 1980). Building on this theoretical account, a recent study found that the three dimensions of vividness – concreteness, emotionality, and proximity – enhanced the persuasiveness of antismoking messages (Ophir et al. 2019). In this study, we focused on concreteness and emotionality. Concreteness enhances vividness by providing detailed and specific information about “actors, actions, and situational context,” and thereby prompts the “imaginability” of the information (Nisbett and Ross 1980, 47). Emotionality is associated with the intensity of imagery induced by news articles that report fact-checking, which contributes to their vividness (Nisbett and Ross 1980; Ophir et al. 2019).

For concreteness, we looked at the information specificity of claims being verified: that is, whether the coverage detailed when (the specific date) and where (the physical space) the claims were made. For example, if the claim was a statement made by a politician during an interview with another news outlet, then the inclusion of the date and the specific occasion (e.g., “in an interview with The Chosun Ilbo on December 26, 2018,” as opposed to “in an interview”) of the claim would enhance the concreteness of the coverage. We also examined the concreteness of verdicts in news articles that reported fact-checking: (a) whether the articles presented a clear verdict (e.g., “the veracity of the claim is debatable” vs. “the claim is found to be false”), and if so, (b) whether the verdict was accompanied by a visual rating scale (e.g., ranging from “true” to “pants on fire”). The number of images used in the article was also expected to enhance vividness by augmenting the imaginability of the article-conveyed information. As an additional indicator of concreteness, we explored whether the articles provided exemplars – detailed personal cases introduced as representations of larger groups of similar cases (Zillmann and Brosius 2000). For instance, an article fact-checking a politician’s remark that unemployment rates were declining could introduce the personal experience of an individual looking for a job. For emotionality – the second dimension of vividness examined herein – we measured the emotional valence (positivity; Trilling, Tolochko, and Burscher 2017) and evocativeness (magnitude; Bright 2016) of the full texts of articles.

In sum, we explored the following message characteristics of news articles that reported fact-checking: (a) how frequently the articles exhibited source transparency and presented contextual information, and (b) how vivid the articles were in terms of concreteness and emotionality (RQ1).

Article Characteristics and Audience Engagement

Audience Engagement

Audience engagement refers to “how involved or responsive the audience is to a particular message” (Humphreys 2016, 48; see also Ha et al. 2018; Steensen, Ferrer-Conill, and Peters 2020). It is a multi-dimensional concept that encompasses the “cognitive, emotional, and affective experiences” of audiences (Broersma 2019, 1) and can be operationalized differently, depending on the goal of the content producer (Napoli 2011). We particularly focused on audience engagement metrics, such as the number of “likes” (Humphreys 2016), associated with each news article that reported fact-checking, because such behavioral data can reflect audiences’ beliefs, evaluations, and emotions experienced through media consumption (Broersma 2019; cf. Steensen, Ferrer-Conill, and Peters 2020).

We examined the following audience engagement behaviors: “like” and “angry” reactions, commenting, and sharing – four prevalent behaviors not only on Naver News, but also on other digital news platforms (Masullo and Kim 2021; Trilling, Tolochko, and Burscher 2017; Weber 2014). These engagement behaviors are apt proxies of the effectiveness of fact-checking coverage because an important goal of fact-checking journalism, correcting misinformation (Amazeen 2020), is only attainable if audiences are exposed to, pay attention to, and ultimately engage with the fact-checking content – accepting its conclusion and participating in further communication. We grouped the four behaviors into (a) evaluative reactions (i.e., “like” and “angry” reactions¹) and (b) post-exposure communication (i.e., commenting and sharing).

Evaluative Reactions

Evaluative reactions tap into the emotional dimension of audience engagement (Broersma 2019; Steensen, Ferrer-Conill, and Peters 2020) and are less public expressions of preference and assessment (Aldous, An, and Jansen 2019). Little is known about the meaning of the “like” and “angry” reactions, which are both executable by clicking on a button placed beneath the news content on Naver News, as is the case on most social media platforms like Facebook. “Like,” as one of the most common forms of audience engagement, may indicate that the audience agreed with, supported, or enjoyed the fact-checking content. However, “like” reactions may also indicate rather thoughtless, intuitive responses to surprising information (Park and Kaye 2021). There is scant empirical research on the signals conveyed by “angry” reactions (cf. Jost, Maurer, and Hassler 2020). The few studies that examined “angry” considered it to mean negative evaluations of the message content (Masullo and Kim 2021). As with “likes,” “angry” reactions at the least, indicate that the audience paid attention to the fact-checking content.

Post-Exposure Communication

Compared to evaluative reactions, post-exposure communication is a relatively public form of engagement (Aldous, An, and Jansen 2019) and may invite subsequent engagement of other audiences by facilitating online discussions or news sharing (Kim 2021). Sharing contributes to increasing the reach of fact-checking content and thereby helps to combat fake news. Commenting, an act of participating in a public “forum” (Tenenboim and Cohen 2015, 200), is also indicative of the success of fact-checking, considering that fact-checking journalism is viewed as a “democracy-building tool” (Amazeen 2020, 99).

Some of the previous studies examined these four behaviors individually (e.g., Xu, Sang, and Kim 2020), while others tested them at once, using a summative index of overall audience engagement (e.g., Gerbaudo, Marogna, and Alzetta 2019). As the current study was a formative investigation of fact-checking coverage, we assessed the effects of its message features on overall audience engagement as a main outcome variable, and additionally conducted exploratory analyses to discern their effects on individual engagement behaviors.

Effects of Source Transparency and Contextual Information on Audience Engagement

Source transparency of fact-checking coverage may engage audiences as it increases the persuasiveness of the articles (Xu, Sang, and Kim 2020). Revealing the evidential sources (e.g., statistics) used for claim verification can enhance the convincingness of the articles’ verdicts by increasing the trustworthiness of fact-checking (Brandtzaeg, Følstad, and Chaparro Domínguez 2018). Moreover, transparency in news articles (e.g., disclosing reporting processes) increases perceived message credibility, which in turn facilitates audience engagement with the articles (Peifer and Meisinger 2021). All in all, fact-checking with greater source transparency may invite more positive reactions, such as “likes,” which, in the setting of the current study, presumably indicate affirmative evaluations of the fact-checking coverage, such as “informative,” “agree,” “important and interesting” (Sumner, Ruge-Jones, and Alcorn 2018). In contrast, it is unclear how source transparency is associated with “angry” reactions, because the interpretation of “angry” reactions associated with fact-checking content carries some ambiguity. Although “angry” reactions connote a negative evaluation of the content (Masullo and Kim 2021), in our context they may mean either (a) disapproval of the fact-checker or the fact-checking process, or (b) negative attitude towards the events and/or person(s) related to the claim being fact-checked. For example, in the latter case, audiences may click the “angry” button even if source transparency has led them to perceive the article to be convincing, to show anger toward a corrupt political figure portrayed in the article. On a different note, source transparency may prompt post-exposure communication (i.e., commenting and sharing). Fact-checking coverage that discloses specific documentation used to support its conclusion would have higher informational utility, which triggers social sharing and word-of-mouth (Berger 2014; Kim 2015).

Contextual information in fact-checking coverage may also invite more evaluative reactions and post-exposure communication, because providing the rationales for the fact-checked claims and discussing the importance of verifying those claims can enhance the argument quality as well as the informational value of the article content. Moreover, news audiences expect good journalism to provide explanations on social issues (van der Wurff and Schoenbach 2014) – a need that contextual information can satisfy. For example, Curry and Stroud (2021) found that the inclusion of background information in news articles (e.g., “why and how a story was written,” 905) enhanced audiences’ intention to engage with the articles (e.g., sharing via social media) by increasing their trust in news organizations.

Effects of Vividness on Audience Engagement

We posit that the more concrete and emotional news articles reporting fact-checking are, the more likely it is that they will foster evaluative reactions and post-exposure communication. A meta-analysis revealed that vividness enhances persuasion, with small-to-medium effect sizes on attitude and behavioral intention (Blondé and Girandola 2016; cf. Frey and Eagly 1993). For example, studies found that employing visuals to boost vividness in tobacco-control messages (Ophir et al. 2019) and fact-checking (Young et al. 2018) increased persuasiveness. Vividness of news articles was also positively associated with share-worthiness (Kim 2015), and the number of comments and depth of discussion in the comment section (Weber 2014).

Specifically, message features that enhance the concreteness of fact-checking coverage, such as information specificity (i.e., when and where the claim was made), conclusion concreteness, exemplification, and images may increase its vividness and hence may trigger audience engagement. First, information specificity – of the claims being verified – may boost the argument quality of the articles, inducing more “like” reactions. At the same time, such information also adds to practical utility, and thereby promotes social sharing of and triggers conversations about the articles (Berger 2014).

Second, as the conclusion (verdict) of fact-checking is an essential component of fact-checking coverage, its concreteness may increase vividness, which in turn may promote audience reception and engagement. Fact-checking coverage that provides an explicit veracity judgment may receive more “likes” from their audiences than coverage that omits the verdict, because the inclusion of an explicitly stated conclusion enhances persuasion (O'Keefe 2002). Moreover, using images alongside the verdicts (e.g., truth scales) may further increase vividness, and thereby generate positive reception from the audiences (i.e., more “likes”; Liu et al. 2017) and facilitate post-exposure communication (i.e., commenting and sharing; Chung 2017).

Third, exemplification may trigger audience engagement. Exemplars may increase the likelihood of receiving “likes” since messages featuring exemplars are more persuasive than those without them (Bigsby, Bigman, and Martinez Gonzalez 2019). Fact-checking coverage that employs relevant exemplars may also prompt user-posted comments and news retransmission because it (a) illuminates a human-interest angle which elevates share-worthiness (Trilling, Tolochko, and Burscher 2017) and (b) conveys the content in a story-like manner which boosts virality (Kim 2015).

Fourth, the emotionality of fact-checking coverage may shape audience engagement (Xu, Sang, and Kim 2020). Audiences’ evaluative reactions may be influenced by the emotional valence of the articles, with positive articles inviting more “like” reactions and negative ones triggering “angry” reactions (Kramer, Guillory, and Hancock 2014). Positive articles are also more likely to go viral, because passing along positive, rather than negative content satisfies motivational needs for promoting the self and helping others, both of which drive message sharing (Cappella, Kim, and Albarracín 2015).

Fifth, emotional evocativeness may invite more “likes” (Xu, Sang, and Kim 2020) due to the high newsworthiness of emotion-laden news stories (Harcup and O'Neill 2001). Emotionally evocative fact-checking coverage may also trigger more audience comments and social sharing. The experience of emotional arousal in response to media messages generates conversations about and retransmission of the messages (Berger 2014; Kim 2015) because of the benefits of such post-exposure communication: helping emotion regulation, strengthening interpersonal relationships, and understanding the media-induced experience (Bright 2016).

Taken together, we explored how (a) source transparency, (b) contextual information, and (c) vividness of fact-checking coverage affected audience engagement behaviors (RQ2).

Method

The study sample consisted of all news articles that reported fact-checking, published online on Naver News between January 2018 and April 2019, by 25 major news outlets in South Korea (N = 914).² The exhaustive list of fact-checking coverage was obtained from an ongoing collaboration among these news organizations, Naver News, and the Institute of Communication Research at Seoul National University. The news articles were content-analyzed using both human and computerized coding methods. The coding items were deductively drawn from the previous research discussed above. Measures of the independent, dependent, and control variables are reported below, along with their descriptive statistics, except for the focal content-analyzed variables which are presented in the results section.

Content Analysis

Human Coding

Human coders content-analyzed the full texts of the articles in terms of (a) source transparency, (b) contextual information, and (c) vividness. For control variables, they coded (d) whether there was a “fact-checking” label in the headline, (e) whether the article examined multiple claims, and (f) the number of subheadings in the full text. The manual content analysis was conducted by two or three coders depending on the coding item. Data used to assess intercoder reliability were randomly drawn from the full news sample (Krippendorff 2013). Intercoder reliability estimates were measured using Krippendorff’s αs and ranged from 0.73 to 1.00 (M = 0.95).³

For source transparency, the frequency with which the articles explicitly mentioned the following evidential sources to support its verdicts was measured: statistics, official reports, laws/rules, news articles, press releases, websites, and interviews. Specifically, the articles’ use of evidence in the process of fact-checking was coded with respect to the seven types of evidence as follows. Numerical data presented were recorded as statistics unless they were part of official reports issued by institutions such as governments, corporations, and universities. To qualify as official reports, mention of either the author or the title of the document was required. Law/rules were coded as present if specific laws, rules, regulations, or public policies were mentioned. News articles were considered to have been employed as evidence if other news reports were cited as evidence. Press releases were recorded if the articles discussed institution-issued documents intended for press circulation. Online sources other than those listed above were coded as websites. Verbatim quotes of person(s) were categorized as interviews if they were employed to verify the claim.

To evaluate the presence of contextual information, coders ascertained whether the news article provided background information, such as an explanation for (a) the reasons behind the occurrence of the claim, and (b) the importance of fact-checking the claim at hand (Salgado and Strömbäck 2012). For the reasons behind the claim, the coders first recorded whether the article identified who made the claims. If the identity of the claimer was present (e.g., a politician whose statement is being verified by the article), the coders decided whether a rationale for the claim was provided. The importance of fact-checking the claim was coded to be present if the article mentioned either the harm inflicted by the claim or the consequences of fact-checking (or not fact-checking) the claim on a societal level. For example, explaining how a widespread rumor about bitcoins is sowing panic among investors would qualify as importance information.

As an indicator of vividness, the news articles were assessed on whether information on when and where the claim occurred (i.e., information specificity) was available. For instance, if the claim under examination was a politician’s statement spoken at a congressional hearing, the date and the title of the hearing would fulfill the time and space components of information specificity, respectively. For conclusion concreteness, coders judged whether the verdicts of fact-checking were provided; and if so, whether the verdicts were presented in just textual formats or alongside visual ratings. For exemplification, the coders recorded whether the articles presented exemplars – article-related specific personal cases used as representations of similar cases (Zillmann and Brosius 2000).

For control variables, coders assessed if the articles were labelled “fact-checking” in their headlines (yes = 61.9%) and whether they examined multiple claims (yes = 15.2%). Lastly, coders also counted the number of subheadings in the article content (M = 1.70, SD = 2.16).

Computerized Coding

To measure the emotional valence and evocativeness of the news articles, we used the Google Cloud Natural Language API (https://cloud.google.com/natural-language/), a pre-trained machine learning API for sentiment analysis. The API analyzed the sentiment of a given text in terms of (a) emotional valence (“sentiment”; the positivity or negativity of the overall sentiment; ranging from −1 to +1) and (b) emotional evocativeness (“magnitude”; the intensity of the sentiment detected, whether positive or negative; ranging from 0 to infinity as it is calculated as a sum of both valences). For each article, we normalized its magnitude score by dividing the score with the length of the article because as the article length increases, the magnitude score – calculated based on the frequency of sentiment occurrences – also tends to increase.⁴

Additionally, we web-scrapped the articles’ full texts and metadata such as news sections and timestamps from Naver News. The articles were published in the following sections: (a) politics (42.6%), (b) national⁵ (30.1%), (c) economy/business (19.5%), and (d) other (e.g., technology, international, culture, sports, entertainment, etc.; 7.9%) topical areas.

By parsing the HTML of the article webpages, we obtained the number of images presented in the articles – this was used as one of the indicators of vividness. Using the HTML parser, we also measured article length (M = 2.14 K Korean characters, SD = 0.96 K). Using timestamps, we calculated the recency of publication – the number of days between the publication date of the earliest article in our sample (1/5/2018) and that of a given article (M = 238.55, SD = 126.08).

We also web-scrapped audience engagement behavior metrics: the numbers of “like” and “angry” reactions, and comments. On each article’s web page on Naver News, audiences could click emoticons, located below the article text, presenting different facial expressions corresponding to “like” and “angry”; they could post comments in the comment section placed below the emoticons; they could also share the article via social media such as Facebook and Twitter. Overall audience engagement was obtained by summing these four indicators. All the above variables were log-transformed given their positively skewed distributions: M = 2.11, SD = 1.62 for “like” reactions, M = 3.49, SD = 2.13 for “angry” reactions, M = 2.49, SD = 1.36 for shares, M = 3.54, SD = 1.98 for comments, and M = 4.54, SD = 1.97 for overall audience engagement. Naver News provided data on (a) the view and share counts for each article and (b) whether the article had been displayed on the main page of Naver (yes = 9.7%).⁶ The view count was log-transformed because its distribution was positively skewed (M = 8.94, SD = 1.90).

Results

Characteristics of Fact-Checking Coverage

Source transparency. Of the seven types of evidential sources, interviews were most frequently employed by the news articles that reported fact-checking (M = 1.08, SD = 1.24), followed by laws/rules (M = 0.77, SD = 1.04). Official reports (M = 0.47, SD = 0.92) and statistics (M = 0.47, SD = 0.95) were less frequently referenced. News articles (M = 0.10, SD = 0.38), press releases (M = 0.05, SD = 0.22), and websites (M = 0.09, SD = 0.34) appeared least frequently in the articles.

Contextual information. About 39.6% of the articles discussed why it was important to verify the claims at hand. About 44.5% provided identifiable claimers and explained why the claims were made, whereas 30.1% mentioned the claimers but not the why information; the remaining 25.4% did not present the claimer identity.

Vividness. Regarding information specificity, 35.4% and 41.8% of the articles indicated “when” and “where” the fact-checked claims were made, respectively. For conclusion concreteness, 26.9% of the articles presented verdicts using visual markers, 57.1% provided only the verdicts, and 16.0% did not provide any verdicts. About 8.2% of the articles employed exemplars. The average number of images per article was 2.12 (SD = 2.08). The average emotional positivity score was −0.01 (SD = 0.05); the average emotional evocativeness was 0.31 (SD = 0.11).

Effects of Article Characteristics on Audience Engagement

Table 1 summarizes the results from linear regression treating the 25 news outlets as fixed effects.

Table 1. Effects of article characteristics on audience engagement behaviors.

	Overall engagement	“Like” reactions	“Angry” reactions	Shares	Comments
Source transparency
Statistics	0.18** (0.06)	0.07 (0.08)	0.28** (0.08)	0.14* (0.06)	0.18** (0.06)
Official reports	0.17** (0.06)	−0.02 (0.08)	0.18* (0.08)	0.11* (0.06)	0.18** (0.06)
Laws/rules	0.04 (0.03)	−0.01 (0.04)	0.07^+ (0.04)	−0.004 (0.03)	0.06* (0.03)
News articles	0.10 (0.09)	0.22^+ (0.12)	0.14 (0.13)	0.05 (0.09)	0.20* (0.10)
Press releases	0.05 (0.11)	−0.18 (0.15)	0.25 (0.16)	−0.13 (0.11)	0.13 (0.12)
Websites	0.04 (0.09)	0.05 (0.12)	0.04 (0.13)	−0.03 (0.09)	−0.004 (0.10)
Interviews	−0.003 (0.02)	−0.02 (0.03)	0.01 (0.03)	−0.02 (0.02)	0.01 (0.02)
Contextual information
Reasons behind the claim
Claimer presented (vs. not)	0.15* (0.08)	0.17 (0.10)	0.23* (0.11)	0.08 (0.07)	0.14^+ (0.08)
Claimer & reasons given (vs. not)	0.18* (0.08)	0.19^+ (0.10)	0.21^+ (0.11)	0.09 (0.08)	0.16* (0.08)
Importance of fact-checking	0.07 (0.05)	0.06 (0.07)	0.11 (0.08)	−0.02 (0.05)	0.15** (0.06)
Vividness
Time: When the claim was made	0.18** (0.06)	0.18* (0.08)	0.30*** (0.09)	−0.09 (0.06)	0.22*** (0.06)
Space: Where the claim was found	−0.01 (0.07)	−0.05 (0.09)	−0.03 (0.10)	−0.07 (0.07)	0.05 (0.07)
Conclusion concreteness
Verdict presented (vs. not)	0.02 (0.08)	−0.14 (0.11)	0.14 (0.12)	0.03 (0.08)	0.08 (0.09)
Verdict w/ visual marker (vs. not)	0.02 (0.08)	0.04 (0.11)	0.05 (0.12)	−0.14 (0.08)	0.13 (0.09)
Images	−0.04^+ (0.02)	−0.01 (0.03)	−0.06* (0.03)	0.002 (0.02)	−0.03 (0.02)
Exemplification	−0.09 (0.09)	0.03 (0.13)	−0.30* (0.13)	0.10 (0.09)	−0.14 (0.10)
Emotional positivity	−0.88 (0.59)	0.55 (0.80)	−1.94* (0.84)	−0.08 (0.58)	−1.68** (0.63)
Emotional evocativeness	0.34 (0.30)	0.24 (0.41)	0.23 (0.43)	0.43 (0.29)	−0.05 (0.32)
N	900	900	900	900	899
R^2	0.87	0.66	0.78	0.74	0.86

Note. Cell entries are unstandardized OLS regression coefficients with standard errors in parentheses. Control variables are not shown here for brevity. Full results are available in Appendix A, supplementary material. All VIFs were smaller than 2.45. Missing data were handled with listwise deletion. ⁺ p < 0.10, * p < 0.05, ** p < 0.01, *** p < 0.001.

Overall audience engagement. Using statistics and official reports as evidence increased overall audience engagement with news articles that reported fact-checking, unstandardized b = 0.18, 95% CI [0.06, 0.29], p = 0.003, and b = 0.17, 95% CI [0.06, 0.29], p = 0.004, respectively. Audience engagement was also higher when the articles provided identifiable claimers of the claims under scrutiny, regardless of whether they additionally explained reasons behind the claims, b = 0.18, 95% CI [0.03, 0.33], p = 0.020, or not, b = 0.15, 95% CI [0.004, 0.30], p = 0.044, compared to when the articles did not specify the claimers. Presenting information about when the claims were made facilitated audience engagement, b = 0.18, 95% CI [0.06, 0.30], p = 0.003. The number of images showed a marginally significant negative association with audience engagement, b = −0.04, 95% CI [−0.08, 0.05], p = 0.084.

“Like” reactions. The use of other news reports as evidence was a marginally significant positive predictor of “like” reactions, b = 0.22, 95% CI [−0.03, 0.46], p = 0.081. Fact-checking coverage that both identified the claimers and provided the reasons for the claims under scrutiny was marginally significantly more likely to induce “like” reactions than coverage with neither, b = 0.19, 95% CI [−0.01, 0.39], p = 0.069. Articles that presented information about “when” the claims were made received more “likes” than those without such information, b = 0.18, 95% CI [0.02, 0.34], p = 0.032.

“Angry” reactions. Referencing statistics, official reports, and laws/rules as evidential sources increased “angry” reactions, b = 0.28, 95% CI [0.11, 0.44], p = 0.001, b = 0.18, 95% CI [0.02, 0.35], p = 0.032, and b = 0.07, 95% CI [−0.001, 0.14], p = 0.053, respectively. Articles presenting the claimers, either with or without additional explanations for why the claims were made invited more “angry” reactions, compared to those lacking the claimer information, b = 0.21, 95% CI [−0.01, 0.42], p = 0.062, and b = 0.23, 95% CI [0.01, 0.44], p = 0.036, respectively – but there was no difference between the articles that included the “why” information and those without it. Articles that mentioned “when” the claims occurred generated more “angry” reactions than those without such information, b = 0.30, 95% CI [0.13, 0.47], p = 0.001. “Angry” reactions decreased when articles (a) presented relevant exemplars, b = −0.30, 95% CI [−0.57, −0.04], p = 0.022, (b) used more images, b = −0.06, 95% CI [−0.12, −0.001], p = 0.047, and (c) conveyed positive content, b = −1.94, 95% CI [−3.59, −0.29], p = 0.021.

Shares. Articles were more likely to be shared when they presented statistics and official reports to support their verdicts more frequently, b = 0.14, 95% CI [0.02, 0.25], p = 0.017, and b = 0.11, 95% CI [0.001, 0.23], p = 0.048, respectively. There was no difference between (a) articles that provided the verdicts either with or without visual markers and (b) those that did not provide the verdicts; but articles presenting visual markers in addition to the verdicts were less frequently shared than those delivering the verdicts alone, b = −0.17, 95% CI [−0.32, −0.01], p = 0.033.

Comments. Articles invited more user comments when they employed statistics, official reports, other articles, and laws/rules as evidence more frequently, b = 0.18, 95% CI [0.06, 0.30], p = 0.004, b = 0.18, 95% CI [0.06, 0.30], p = 0.004, b = 0.20, 95% CI [0.01, 0.40], p = 0.039, and b = 0.06, 95% CI [0.004, 0.11], p = 0.036, respectively. Users posted more comments when the claimers were discernible, regardless of whether the rationales for the claims were additionally given, b = 0.16, 95% CI [0.001, 0.302], p = 0.049, or not, b = 0.14, 95% CI [−0.02, 0.30], p = 0.076, compared to when no claimer information was provided.

Articles generated more user comments when they (a) discussed why it was important to fact-check the claims under examination, b = 0.15, 95% CI [0.04, 0.26], p = 0.007, (b) specified when the claims were made, b = 0.22, 95% CI [0.09, 0.34], p = 0.001, and (c) delivered negative content, b = −1.68, 95% CI [−2.90, −0.45], p = 0.008.⁷

Discussion

We examined (a) what message variations characterized news articles that reported fact-checking and (b) how those message features shaped audience engagement with the articles. We content-analyzed fact-checking coverage from 25 major news outlets in South Korea (N = 914) using both manual and computerized coding procedures. Next, we combined the content-analysis data with associated behavioral data of audience engagement on Naver News, including “like” and “angry” reactions, shares, and comments to identify patterns of associations between article characteristics and audience engagement behaviors. An important methodological contribution of this study is that we controlled for view count associated with each article when predicting the frequency of audience engagement behaviors that can mostly only be undertaken after viewing the content, and thereby addressed the missing denominator problem (Tufekci 2014) prevalent in studies using digital trace data of user activities.

The results showed that source transparency with respect to statistics and official reports promoted audience engagement behaviors with news articles reporting fact-checking (except “like” reactions), suggesting that transparency is not only an important principle in fact-checking (Humprecht 2020), but also a significant driver of evaluative reactions and post-exposure communication. Overall, the finding is consistent with previous research documenting positive effects of disclosing reporting processes on audience engagement (Peifer and Meisinger 2021). At the same time, the result also raises an important question: why was the use of statistics and official reports particularly effective? One viable explanation concerns audiences’ receptivity to numbers: Presentation of numerical information in a news article may increase perceived accuracy, regardless of its actual veracity (van Dijk 1988).

It is also noteworthy that the two forms of evaluative reactions (“like” & “angry”) showed quite different relationships with the use of specific evidential sources. Using statistics and official reports did not affect “like” reactions, but it increased “angry” reactions. Relatedly, an ancillary analysis revealed that the two evaluative reactions were positively associated with the numbers of shares and comments, over and above all the predictors used in the main analysis (see Appendix A, supplementary material), b_like = 0.18, se_like = 0.02, p < 0.001, b_angry = 0.12, se_angry = 0.02, p < 0.001 for shares; b_like = 0.25, se_like = 0.02, p < 0.001, b_angry = 0.50, se_angry = 0.02, p < 0.001 for comments, which is consistent with a previous study (Larsson 2018) that identified the same pattern of relationships between (a) “like” and “angry” reactions and (b) shares and comments using the data from Facebook pages of Scandinavian newspapers. These results, combined with the abovementioned finding that the use of statistics and official reports was positively associated with the numbers of comments and shares, suggest that the emotional experience of anger, but not “like,” might mediate the effects of the evidential sources on these post-exposure communication behaviors. However, our data are not well-suited to ascertain whether the relationships between the evaluative reactions and post-exposure communication behaviors are causal. Future research should therefore experimentally test how “like” and “angry” reactions differentially mediate the effects of source transparency on sharing and commenting.

In contrast, the frequency of interview sources, while most prevalently used in fact-checking coverage, was not associated with audience engagement. One explanation concerns the unique expectations that audiences may have for fact-checking, in comparison to more traditional genres of journalism. At the heart of fact-checking journalism is the idea of “transparency of sources” which can be realized by using primary, on-the-record sources, as exemplified by the International Fact-Checking Network’s code of principles (http://poynter.org/ifcn/). This emphasis on independent, in-depth investigation into claim verification is an overhaul from traditional journalism’s heavy reliance on interviews (Gans 1979), a source that audiences cannot trace back to.

Consistent with previous research (Curry and Stroud 2021), the presence of contextual information in fact-checking coverage was an important predictor of audience engagement. Identifying the speakers of the claims under scrutiny increased “like” and “angry” reactions, and comments, but the additional provision of reasons for the claims did not have incremental effects. We suspect that the presence of an identifiable claimer increased vividness to a significant degree, but additional contextual information did not add much to evoke imagery. Alternatively, it may be that audiences paid greater attention to factual information such as the identity of the claimer, compared to a more interpretive one like the journalist’s speculation on the intentions/motivations of the claimer. This account seems to be supported by our finding that presenting information about when the claim was made, another kind of factual information, also triggered “like” and “angry” reactions, as well as audience comments. Future research will need to experimentally test the psychological processes underlying the observed effects.

Articles invited more audience comments when they discussed why it was important to fact-check the claims at hand, suggesting that providing richer contextual information helps audiences better understand the issues addressed by the articles, and thereby facilitates active participation (Curry and Stroud 2021). In other words, contextual information in the news may operate as a significant ingredient that “fuels conversation” (Katz 1992, 80), given the dialogical nature of commenting on news articles. Interpreting the implications of audience comments on the goals of fact-checking, however, requires a more nuanced approach. For instance, if audience comments discredit the fact-checker or reiterate the claim found to be false, they contribute to amplify, rather than reduce, misperceptions (Lee, Kim, and Cho 2017). Additional investigation into how audience comments influence other members of the audience is warranted.

Among the message features related to the concreteness (i.e., one of two dimensions of vividness examined) of fact-checking coverage, the mention of “when” the claim under scrutiny was made had consistent positive effects on audience engagement behaviors (except for sharing), highlighting vividness as an important determinant of audience engagement.

We also observed a notable difference in how specific message features pertaining to concreteness affect “angry” reactions. Whereas articles providing information about when the claim at hand was made triggered “angry” reactions, those presenting exemplars and images were less likely to do so. One possible explanation is that these vividness-related features may differ in their congruency with the main theme of the fact-checking coverage (Smith and Shaffer 2000). Providing factual details about the claim under evaluation may strengthen the persuasiveness of the article reporting fact-checking, whereas using exemplars and images may undermine it by distracting the audience from what the article intends to convey – that is, the veracity of the claim. Indeed, Frey and Eagly (1993) found that vivid editorials employing exemplars, metaphors, and embellishing adjectives were less persuasive than their pallid counterparts, because vivid components distracted people from the core theme of the messages (see also Smith and Shaffer 2000). Another possible explanation concerns the information processing of the article (cf. Ophir et al. 2019). Exposure to exemplars and images may lead audiences to process the article via a more experiential (vs. cognitive or analytical) route which entails a greater level of absorption, making the audiences less likely to respond negatively (Green 2006), compared to when processing factual details related to the claim under scrutiny. Future research might need to experimentally test how people process various vividness-related message characteristics to uncover the mechanisms that underlie the observed patterns herein.

More research is also warranted to examine the effects of the abovementioned message features of news articles other than those reporting fact-checking, considering the previous studies that have pointed to the distinct characteristics of the fact-checking audience: political liberals have more favorable attitudes towards fact-checking than conservatives do (Robertson, Mourão, and Thorson 2020). It would be interesting to investigate whether the observed relationships between message features and audience engagement behaviors hold for other genres of journalism.

The relatively low explanatory power of our focal message characteristics for audience news sharing deserves mention. Previous research shows that messages with high informational utility – which in the context of our study are news articles that report fact-checking with accessible evidential sources, details about the claims under scrutiny, among others – are more likely to go viral. Except for statistics and official reports, however, none of the focal content features were significantly associated with news sharing. At first blush, this result seems to contradict previous findings on message virality, but the particular setting of the current study might help make sense of it. Audiences might have thought that sharing fact-check-worthy, controversial news would disclose their own opinions and discomfort their recipients – especially those with opposing views (Cappella, Kim, and Albarracín 2015). Our data support this line of reasoning: sharing was a relatively less frequent audience engagement behavior than “angry” reactions and comments. In other words, the controversiality – and often the politically polarized nature – of fact-checking may have inhibited audience sharing across the board. The result that shareability-enhancing effects were found only for statistics and official reports can be explained by the self-presentation and altruistic motivations that drive message sharing (Berger 2014). That is, audiences may have considered statistics and official reports more objective (and hence more useful) than single-case evidence such as interviews, and thus expected sharing news articles that reported fact-checking with such useful pieces of information to both help the recipients and place the sharers themselves in a positive light, despite the (potential) controversiality of the content. In this sense, the result is consistent with previous research pointing to informational utility as an important driver of news sharing (Kim 2015). That commenting was a more frequent mode of audience engagement also supports this line of thinking: people may have been more willing to post comments below the fact-checking content because those comments are less likely to be seen by their social networks.

Our message characteristics also had a relatively low explanatory power for “like” reactions. None of the focal message features was statistically significantly associated with “likes,” except for the time component of information specificity (i.e., when the claim was made). Considering that a wide array of content characteristics was examined here, one possible explanation is that people might click the “like” button in a rather thoughtless manner (Park and Kaye 2021), rendering the magnitudes of its relationships with message features negligible. Or it could be that individuals hold varying motivations for “liking” fact-checking coverage. Because people attach different meanings to “likes,” some may find a message feature to be a strong reason to “like” a given article, whereas others may consider it an equally strong reason to not conduct the behavior. This variability might have made it unlikely for a message feature to have, on average, a statistically significant effect on the number of “likes.” In contrast, “angry” reactions were better explained by our focal message features, suggesting that people may share a much clearer idea of what it means to click the “angry” button. The divergent patterns of associations between the “like” and “angry” reactions, as well as between commenting and sharing underscore the need for further theorizing what message features shape each individual audience engagement behavior.

All in all, this study contributes to the literature by documenting not only how message characteristics of fact-checking coverage influence overall audience engagement, but also how they relate to specific individual engagement behaviors differently. The results of this study should be interpreted considering several limitations. First, the news articles that reported fact-checking were collected in South Korea. Thus, our results may not necessarily generalize to other countries.

Second, we explored the relationships between message features of news articles and audience engagement with the articles using population-level news content and audience behavioral data. While this observational design allowed us to capture the associations that exist in a natural setting, given the lack of random assignments to manipulated message characteristics, we cannot conclusively rule out the possibility that the observed relationships are spurious. For example, history factors such as specific political or social events that occurred during the data collection period might explain the relationships. Future research should also examine other message features, including news values (Harcup and O'Neill 2001) that have been shown to influence audience engagement behaviors such as sharing (Trilling, Tolochko, and Burscher 2017) and commenting (Weber 2014).

Relatedly, third, since we used news article-level, rather than audience-level, data, the question of why the observed patterns of associations occurred – that is, what individual-level processes underlay the observed message-level patterns – remains an unanswered question. We provided speculations about some possible social psychological mechanisms. Future research will need to conduct experiments to examine those and other potential processes of the observed effects of article content features on audience engagement.

Fourth, we used the Google Cloud Natural Language API to quantify the emotional valence and evocativeness of the news articles. Google’s API enables researchers to make predictions about sentiments using pre-trained models. However, these models are black boxes that do not offer explanations for how the sentiments are detected and weighted. Therefore, to examine and more ultimately improve the validity of our sentiment analysis, future work should employ other machine learning and lexicon-based text analysis tools.

Fifth, our data did not allow a thorough investigation of audience engagement. Other important dimensions of audience engagement, such as spatiotemporal and normative dimensions (Steensen, Ferrer-Conill, and Peters 2020), could not be addressed by the data which focused exclusively on manifest behaviors (Broersma 2019). While this study can contribute to our understanding of how message features affect audience engagement, it remains incomplete without a more comprehensive examination of the very notion of audience engagement.

Fact-checking is an important practice of journalism in the emerging media environment where misinformation and fake news proliferate rapidly and widely. Fact-checking journalism cannot accomplish its goal to correct misinformation and fight fake news unless it invites a certain level of engagement from news audiences. Scientific knowledge on (a) which message features characterize fact-checking coverage and (b) of these, which engage audiences will ultimately improve journalistic fact-checking by enhancing its effectiveness.

Acknowledgements

The authors acknowledge the funding support of the Institute of Communication Research at Seoul National University, and the Ministry of Education of the Republic of Korea and the National Research Foundation of Korea (NRF-2018S1A5B8070398). The content is solely the responsibility of the authors and does not necessarily represent the views of the funding agencies.

Disclosure Statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

The authors acknowledge the funding support of the Institute of Communication Research at Seoul National University, and the Ministry of Education of the Republic of Korea and the National Research Foundation of Korea (NRF-2018S1A5B8070398). The content is solely the responsibility of the authors and does not necessarily represent the views of the funding agencies.

Notes

1 Naver News also allowed other evaluative reactions such as “sad” and “heartwarming.” We opted not to test these reactions because (a) they were not allowed in many other news platforms and (b) they were much less frequent than “like” and “angry” reactions (M = 2.94, Mdn = 0 for “sad”; M = 1.58, Mdn = 0 for “heartwarming”; M = 45.46, Mdn = 5 for “like”; M = 254.36, Mdn = 34 for “angry”).

2 The 25 news outlets were: The Asia Business Daily, The Chosun Ilbo, The Korea Economic Daily, The Segye Times, The Seoul Shinmun, Chosun Broadcasting Corporation (TV Chosun), Donga Science, Edaily, Hankook Ilbo, Inews24, Joongang Ilbo, JTBC, Korean Broadcasting System (KBS), Maeil Broadcasting Network (MBN), Maeil Business Newspaper, Money Today, Munhwa Broadcasting Corporation (MBC), Munhwa Ilbo, News1 Korea, Newsis, Nocut News, Ohmynews, Seoul Broadcasting System (SBS), Yonhap News Agency, YTN.

3 Krippendorff’s αs were greater than 0.80 for all coding items, except exemplification (α = 0.73). Exemplification-related results should therefore be interpreted with caution considering this relatively low intercoder reliability.

4 The sentiment of the claims being verified was weakly positively associated with that of the article full texts (which included the claims), r = 0.21 for emotional valence and 0.13 for emotional evocativeness (ps < 0.001). The articles’ sentiment was unaffected by the direction of their verdicts (truth, false, or neither; n = 768 articles that provided their verdicts).

5 The “national” topic included domestic news on various issues in South Korea, such as crime, public safety, and education – topics that are, strictly speaking, not related to politics or economy/business.

6 The audience engagement data were collected on May 27, 2019.

7 Appendix B, supplementary material, reports how the topic areas of fact-checking coverage relate to (a) its message characteristics and (b) audience engagement.