What Drives Perceptions of Foreign News Coverage Credibility? A Cross-National Experiment Including Kazakhstan, Russia, and Ukraine

ABSTRACT

Research on news credibility and susceptibility to fake news has overwhelmingly focused on individual and message-level factors explaining why people view some news items as more credible than others. We argue that the consistency of the message’s content with the dominant mainstream narrative can have a powerful explanatory capacity as well, particularly in the domain of international news. We test this hypothesis experimentally using a sample of 8,559 social media users in three post-Soviet countries. Our analyses suggest that the consistency with the dominant narrative increases the perceived credibility of foreign affairs news independently of their veracity. We also demonstrate the moderating role of international conflict, government support, and news language in some national contexts but not others. Finally, we report how the effects of these factors on credibility vary according to whether the news items are real or fabricated and discuss the societal implications of our findings.

KEYWORDS:

• News credibility
• international news
• online experiment
• political communication
• strategic narratives
• rally-round-the-flag

View correction statement:

Correction

Introduction

In recent years, the growing scholarly and public concern over the spread of online misinformation has reinvigorated the area of communication research examining the determinants of people’s susceptibility to fake news that, in turn, correlates with their perceptions of information credibility. Credibility – the news item’s quality of being believable or trusted – is a multi-level construct that may reflect individuals’ attitudes toward the message source, the content of the message itself, or the media in general (Appelman & Sundar, 2015; Metzger et al., 2003).

The central objective of this study is to examine how the effects of dominant and alternative news narratives on the perception of message credibility by social media users are moderated by important political factors, above all – by the presence of international conflict, perceived quality of the international relationships, and individual governmental support. Past research has focused on two groups of factors: message features and individual features. The latter includes their psychological, social, and political attributes (Bryanov et al., 2021; Tandoc, 2019). In this paper we argue that factors at the level of national media systems, such as nationally dominant media narratives, and international relations, such as existence of bilateral conflict, may affect news credibility independently of individual political predispositions. We show that these factors are particularly important in foreign affairs coverage.

Foreign affairs news is the domain of media coverage where information consumers are most likely to be exposed to uniform and stereotypical coverage (Galtung & Ruge, 1965). Compared to domestic news that is of more immediate interest, people tend to be less motivated and often ill-equipped to learn what is happening abroad (S. E. Bennett et al., 1996). Furthermore, across different media systems, foreign affairs coverage is heavily shaped by governments’ strategic interests and is limited in scope (Aalberg et al., 2013). The latter may be a result of domestically uncontested governmental pressure, but also of media overreliance on official sources and largely shared interpretations of national interest (Bennett, 1990). Paradoxically, except for a recent study by Erlich and Garner (2023), little research has been done on message credibility using foreign affairs news.

In this study, we experimentally confirm the existing theories by showing that the major driving force of foreign affairs news credibility is its concordance with a narrative that dominates a given national media system, rather than its veracity. More importantly, we show this effect to be more pronounced in news covering the country whose relations with the respondent’s country are tense, and to be distinct from relevant individual political predispositions – namely, from the individual’s support for their government and their perception of international relations with the country covered in the respective news item.

The moderating effect that the two latter factors have on the relationship between type of narrative and news credibility represents the main theoretical contribution of our research. We demonstrate not only the importance of country-level dominance of certain discursive patterns in shaping trust of international news, but also show what amplifies and limits this trust and, specifically, the role of international conflict for trust formation.

Our methodological contribution is the application of an innovative cross-country experimental design that has never been used in news credibility research. In addition, we employ linear mixed modeling with an orthogonal set of contrasts and related interaction terms. This allows us to test a complex theory of news credibility in a single model, avoiding overestimation of separate predictors. Finally, this is the first comparative study on news credibility carried out in the post-Soviet space – a region whose global political significance has become apparent in 2022.

Historical Context and Case Selection

To fill the gap in research on message credibility, we employed a cross-country online experiment that took place in 2020 and involved 8,559 participants from three countries compared pairwise: Russia vs Ukraine and Russia vs Kazakhstan. Here, we briefly describe the regional context to facilitate comprehension of our hypotheses.

Relations of former colonial or imperial powers with their colonies present a special case that has not been addressed in news credibility research. Although the Soviet Union has never been a classical colonial power, it was nevertheless divided into the dominant and the dependent territories, and after its dissolution, the relations between these territories have developed in different ways, similarly to those between former colonial powers of Europe and their colonies (Etkind, 2011). However, unlike the latter, former Soviet republics have stayed neighbors with Russia, and most of them still share historical memories, cultures, and strong cross-border ties, therefore, they had to establish relations irrespective of how different their interests have become.

The selection of Russia, Kazakhstan, and Ukraine for our research was informed by this context. While the three countries share many similarities, there are major differences in the structures of their respective media systems and bilateral relationships. By the time of the data collection in early 2020, Russia and Ukraine were already in a state of intense political confrontation, but the Russia-Kazakhstan relationship was conspicuously friendly. Thus, the two pairs exemplified sharply contrasting cases in terms of the presence/absence of international conflicts, while the Ukrainian-Kazakhstani pair was neutral. Therefore, in this study we did not examine news perception in this latter pair.

There were apparent differences between the three countries’ media systems at the time of our research. In Russia, the government enjoyed consolidated control over the mainstream media (Kiriya, 2019) and maintained a substantive online presence. This combination allowed state actors to ensure that the government’s strategic narrative largely overlapped with that of the mainstream media, which has become especially conspicuous after the escalation of tensions with Ukraine over Crimea and Donbass in 2014 (Szostek, 2017; Szostek & Hutchings, 2015).

In contrast, in Ukraine, where the news landscape was more fragmented, no political entity maintained full control over the dominant narrative around the nation’s relationships with its neighbors (Korbut, 2021). However, by 2020 the protracted military standoff with pro-Russian forces in the country’s Eastern regions fueled patriotic consolidation in public discourse, marked by acute anti-Russian overtones (Szostek, 2018). Independently of any single actor’s strategic efforts, the mainstream media narrative around foreign policy in Ukraine incorporated a variety of discourses that were highly critical of Russia.

A different situation was observed in Kazakhstan: the government wielded significant influence over the media landscape (Schatz, 2009), nevertheless, the country was not involved in a confrontation with Russia. As we elaborate further below, this might have reduced the need for the nation’s political elites to invest heavily in strategically shaping the foreign policy narrative.

Conceptual Framework

In this section we (a) review the concept of news credibility and studied factors affecting it, (b) explain the concept of strategic narrative and its relevant subtypes (dominant and alternative), and (c) differentiate between the dominant and pro-government narrative.

News credibility research has a long tradition (Appelman & Sundar, 2015; Metzger et al., 2003), where the concept itself is commonly understood either as an objective quality of a news item that can be measured based on its features or as a subjective consumer’s judgment about message trustworthiness. The latter is also referred to as perceived credibility, which is the focus of this paper (Mena, 2020; Schaewitz et al., 2020). In conceptualizing perceived credibility, Appelman and Sundar (2015) place message credibility between several related superconcepts, including source credibility and media credibility, and subconcepts, such as the message’s quality of being error-free, unbiased, and consistent.

Traditionally, credibility research has developed separately from research on misinformation and deception in news, with only truthful news items being used as stimuli. Moreover, credibility has often had positive connotations, as a feature that media professionals should strive for (Henke et al., 2020). Only recently have scholars discovered that the same factors that decrease the perceived credibility of fakes may also decrease the credibility of true news, and vice versa. Thus, an intervention in the form of a general warning (Clayton et al., 2020; Tandoc et al., 2021) or a lower number of likes (Luo et al., 2022) were linked to a decrease in the credibility of both fake and true news. When describing the adverse effects of fake news on media credibility, Tandoc Jr. et al. (2021) argued that fake news is “poisoning the information well” by undermining the public’s trust in journalism as a profession and news media as a social institution.

Bryanov et al. (2021) documented that most research that explored the predictors of accuracy in fake news detection by humans measured perceived message credibility. A few studies showed a significant interaction between news item veracity and other factors in their effects on message credibility (Bronstein et al., 2019; Pennycook & Rand, 2019). This makes control for message veracity important for credibility research.

The factors that are known to affect perceived message credibility first include news item features, such as the presence of scientific sources, statistical information, and their visualization (Henke et al., 2020), hyperlinks (Borah, 2014), or “gamified” news style (Hopmann et al., 2015). Second, they include environmental features, such as the presence of online reader comments (Conlin & Roberts, 2016), the negative valence of the comments (Kluck et al., 2019; Waddell, 2018), presence of recommendations from online opinion leaders (Mena, 2020; Turcotte et al., 2015), the number of likes (Luo et al., 2022), and different features of news sources (to be addressed in more detail further below). Finally, the third group of factors is the interactions of the characteristics of individual news consumers with news features, such as news consistency with consumers’ prior beliefs (Moravec et al., 2018; Pennycook & Rand, 2019) and prior exposure (Pennycook et al., 2018). However, the effects of higher-level concepts, such as strategic narratives manifested in messages, have not yet been investigated.

Strategic narratives are frameworks that allow people to connect social phenomena into structured, comprehensible storylines (Freedman, 2017; Miskimmon et al., 2014). These narratives are “strategic” because they are crafted by political actors to advance desired interpretations of international relations. According to Miskimmon et al. (2014), narratives are conceptually related to discourses and frames as parent-to-child concepts. They are dynamic storylines that coherently explain a particular area of international affairs by incorporating different frames and discourses. Depending on the degree of power consolidation in each media system, strategic actors have varying capabilities to shape narratives (Miskimmon et al., 2014). As Krebs (2015) notes, narratives operate within the boundaries of a broader national culture; they are both informed and constrained by the values and worldview of their target audience (Wilkinson & Gow, 2017). That is, political actors incorporate the existing constraints and possibilities into their strategies of narration.

Although these actors may be in competition with one another, some of them may have more power than others to make their strategic narratives dominant (Gurevitch & Levy, 1985). The heavy prevalence of one narrative is common for societies with centralized control over the media and for highly mobilized societies experiencing the rally-around-the-flag effect (Barnett & Roselle, 2008; Baum, 2002; Entman, 2004; Groeling & Baum, 2008). Additionally, as outlined in the introduction, the asymmetric dominance of one narrative is particularly pronounced in foreign affairs news where political elites have an informational advantage (Baum & Zhukov, 2019; Entman, 2004) and enjoy their interpretations of reality being taken as “common sense” (Huysmans, 1998; Milliken, 1999).

Therefore, by dominant narrative, we understand a coherent system of interpreting international affairs that is prevalent in a given national media system. Here, the idea of “domination” represents both a sense of stability and continuity in foreign news coverage and the agenda-setting dimension of the dominant narratives (Wanta et al., 2004). These narratives are activated by the elites. Mainstream media channels then adjust their news coverage accordingly and make sure that journalists and pundits convey a coherent and unified interpretation of world affairs to the public (Entman, 2004). Accordingly, we define alternative narratives as distinct interpretations of international affairs that explicitly challenge the dominant “mainstream” narrative. Here, our definition emphasizes contextual and relational qualities of alternative narratives that oppose “the overall tendency of public discourse emanating from what is perceived as the dominant mainstream media in a given system” (Holt et al., 2019, p. 862).

Theoretically, one can imagine situations when dominant narratives are distinct from those of the political elites, but empirically in the domain of international affairs, they largely overlap, while alternative views are marginalized (W. L. Bennett et al., 2007). Dominant narratives may also be distinct from the dominant public opinion, as in highly controlled societies narratives prevailing in the media may be very different from the political views of the majority. Likewise, narratives are distinct from individual political views. In media systems where the audiences are largely disengaged and uninformed (Baum & Zhukov, 2019), opinions on international affairs are usually simplified and do not represent the entire political spectrum. Rather they vary along two simplified lines: subjective placement of the relations with a given foreign country on a friend-foe continuum and the degree of agreement with the dominant narrative on that country. As dominant and government narratives are likely to intersect, it is also likely that the perceptions of foreign countries will correlate with government support; in any case, these two dimensions of political views are important to be accounted for when studying the effects of narratives on message credibility.

Hypotheses and Research Questions

Based on the conceptual framework introduced above, we outline our hypotheses starting with the simple effect of narrative on news credibility and then focusing on its nuanced interactions with other conceptually important factors. Although the effect of narrative is highly expectable, it has never been documented via rigorous experimental research, and its size, as compared with other factors, in unknown. Additionally, we expect the mechanism by which news narrative affects news credibility to differ from the effect of the message’s political alignment in more symmetric political contexts (e.g., Democrat vs Republican stances in the US domestic affairs). In the latter context, what would matter is the concordance of this orientation with the consumer’s political views, which has been widely studied (Hart et al., 2009; Osmundsen et al., 2021). However, when one narrative heavily dominates, its effect will be more similar to that of the spiral of silence (Noelle-Neumann, 1991): the awareness of the dominant way of representing a foreign country will exert cognitive pressure on news consumers and make them perceive the events covered from the dominant perspective as more plausible (Entman, 2004). That is, although political views, and especially government support may moderate the effect of narrative on credibility, we nevertheless expect the narrative to have its own strong effect, independent of individual political views:

H1: Foreign affairs news items representing a dominant narrative will be perceived as more credible than those representing an alternative narrative across all countries.

More importantly, we expect that this effect will be influenced by the relationship between the respective countries. In particular, the effect might be stronger if the country covered in the news and the country to which a news consumer belongs are in conflict. The first mechanism behind this is that conflicting governments may have additional incentives for mobilizing domestic support by strategically manipulating news. A related mechanism at play may be the “rally-round-the-flag” phenomenon, whereby in times of international crises, both the press and the public tend to be more willing to accept patriotic rhetoric (Barnett & Roselle, 2008; Baum, 2002; Groeling & Baum, 2008). Thus, we expect that:

H2: The effect of narrative type on news credibility will be greater for respondents from countries that are in a state of conflict seeing news about each other’s countries, as compared to respondents from countries that are not in conflict seeing news about each other’s countries.

As noted above, two dimensions of political views are important in this context. The first is the perceived degree of conflict in the relations between the country covered in the news and the country of the news consumer. We use it as a control variable to differentiate between narrative as a system-level effect and the relevant political views as an individual-level effect. The second dimension is the degree of government support used in country-specific hypotheses. We expect that the effect of narrative should be moderated by the user’s government support where the intersection of the dominant and the government narratives is the strongest. Thus, we expect the interaction to be weaker in Ukraine where media power is relatively fragmented (Orlova, 2016; Ryabinska, 2011). At the same time, even though media control has been relatively centralized in Kazakhstan, we do not anticipate this effect there. The government of Kazakhstan has had fewer incentives to instrumentalize media in their coverage of Russia due to the absence of conflict between these two countries (Anceschi, 2020). However, in Russia, both media control centralization and instrumentalizing incentives are expected to be strong. Therefore, our next hypothesis is the following:

H3: Greater government support is associated with a greater effect of narrative type on perceived credibility for Russian respondents, but not for respondents residing in Ukraine or Kazakhstan.

We further develop this hypothesis by focusing on the interactions involving narrative type, news consumer country, and the degree of government support. Here, we make two specific predictions at the level of simple interactions:

H3a: Greater government support is associated with higher perceived credibility of news items representing a dominant narrative for Russian respondents, but not for respondents residing in Ukraine or Kazakhstan;

H3b: Greater government support is associated with lower perceived credibility of news items representing an alternative narrative for Russian respondents, but not for respondents residing in Ukraine or Kazakhstan.

In Ukraine, where the majority of the population are Ukrainian-Russian bilinguals, language choice is often motivated by political considerations. They are largely shaped by Ukraine’s post-Soviet nation-building trapped between pro-European orientations dominant in the West of the country and more pro-Russian orientations that were still common in the East in 2020 (Pop-Eleches & Robertson, 2018; White & Feklyunina, 2014). These orientations correlated with the language that Ukrainian residents considered their mother tongue (see also, Erlich & Garner, 2023). Therefore, in our experiment where Ukrainian participants had an option to select the language of news items shown to them, those who opted for the Ukrainian language might have done that because of their more patriotic, anti-Russian stance. This stance, as we demonstrate in subsequent sections of this paper, is aligned with the narrative that dominated coverage of Russia in the Ukrainian media. Accordingly, we hypothesize that:

H4: For Ukrainian respondents, the effect of narrative type on news item credibility is higher if the respondent chooses to read the news in the Ukrainian rather than the Russian language.

As was mentioned earlier, news veracity is an important factor that can both affect news credibility directly and interact with other factors influencing the latter. While true news items are, on average, rated as more credible than fake news (see, e.g., Erlich & Garner, 2023; Pennycook & Rand, 2019), the existing literature does not provide theory-driven directional expectations regarding the interactions of news veracity with the factors of our interest. Indeed, different from earlier research, we did not use extant fake news, but constructed our experimental stimuli such that they were very similar to true news to minimize the contribution of knowledge to effects of narrative, state of conflict, and individual-level dispositions on the credibility ratings. Therefore, we pose only a broad exploratory research question:

RQ1: To what extent do the effects of narrative type, conflict presence, and language on news credibility differ depending on the veracity of a news item?

Likewise, existing results on the effects of news sources are inconclusive: while source credibility has been consistently shown to be positively related to message credibility (Kim & Dennis, 2019; Srinivasan & Barclay, 2018), the effects of source types have been different: from null (Clayton et al., 2019; Tsang, 2021) to the opposite from the predicted direction (Tandoc, 2019) to complex interactive effects (Karlsen & Aalberg, 2021). Since there is no research on the effects of the country of the source, we pose an additional exploratory research question:

RQ2: Will news about a foreign country attributed to a domestic source be perceived as more or less credible than news about a foreign country attributed to a source from that country?

Materials and Methods

Narrative Discovery and Operationalization

To discover and describe discursive elements of narratives (such as frames and tropes), we build on previous work that used computational and qualitative methods in identifying systematic patterns in Russian, Ukrainian, and Kazakhstani mainstream media’s coverage of the neighboring nations (Koltsova & Pashakhin, 2020; Kazun & Pashakhin, 2021; Vziatysheva et al. 2021).

The workflow proceeded as follows. We collected separate corpora of all news texts produced by the thirty most popular online media organizations in each of the three countries between January 2018 and June 2019. We used the data of Brand Analytics and Mediaologia market analytics companies to evaluate the popularity of sources, while regional experts checked the political representativeness of these source samples.¹ Four narrower collections were formed from these sources: Russian news about Ukraine (~286,000 texts), about Kazakhstan (~42,000 texts), news about Russia produced by Ukrainian media (~23,000 texts) and by Kazakhstani media (~22,000 texts). These corpora were divided into thematic clusters by a Gibbs-sampling-based LDA topic-modeling algorithm (Griffiths & Steyvers, 2004; i.e., a soft clustering technique allowing multiple topics in a single text). Then, guided by topic modeling, we analyzed each corpus to identify sets of topics that dominated the news agenda in Russia, Ukraine, and Kazakhstan regarding each of the target foreign nations. For each country, a regional media expert supervised the interpretation of relevant topics. These experts also suggested additional topics that had not been captured by the automated analysis. Next, the experts manually identified the prevalent ways of narrating the stories in the sample of texts within each topic, as well as those narratives that challenged the former. Based on the observed patterns, we formulated a series of statements that described either a dominant or alternative view on a particular issue (e.g., “the Eurasian Economic Union is a boon for Kazakhstan’s economy” vs. “Russia benefits from the Eurasian Economic Union while for Kazakhstan it is a burden”). We then collected or constructed news items that matched these statements. Finally, the three experts reviewed all news items to make sure they matched the selection criteria.²

News Item Construction

We constructed four sets of items, grouped in two pairs: (1) News about Ukraine shown to the Russian audience and news about Russia shown to the Ukrainian audience; (2) News about Kazakhstan shown to the Russian audience and news about Russia shown to the Kazakhstani audience. We produced separate sets of news items about Russia for Ukrainian and Kazakhstani users to fit country-specific narratives. Each set of news items consisted of 24 items varying in the narrative and veracity (see Appendix B and Supplement A for examples). This resulted in 72 unique news items in Russian, of which 24 news were also translated into Ukrainian, thus forming a total of 96 items.

Veracity: true/fake. We obtained real news from actual media sources and fact-checked them in at least two independent publications. We chose to generate fake news ourselves because it proved impossible to find “real” fakes (such as debunked false news aggregated by dedicated websites) fitting our requirements, especially those that would represent dominant and alternative narratives. All fake news items were generated by a professional journalist.

Narrative represented: dominant/alternative. News items were either collected or constructed to represent dominant and alternative narratives that had been identified according to the described above procedure. All items were validated by media experts from Russia, Ukraine, and Kazakhstan.

News language. Ukrainian users could choose between Russian and Ukrainian language of news and of the experimental interface. A native Ukrainian-speaking scholar translated original Russian-language stories into Ukrainian and a Ukrainian media expert then proofread them. For Russian and Kazakhstani users news were shown in Russian only. Even though the Kazakh language is gradually gaining popularity in Kazakhstan, “Russian still appears to be the language of preference for most media consumption” (Emrich et al., 2013, p. 24). We discuss the ramifications of not showing the news in the Kazakh language in the Limitations section.

News source. News items were randomly attributed to either domestic media (e.g., when Russian respondents read the news about Ukraine from Russian media) or the media of the country covered in the news item (e.g., when Russian respondents read the news about Ukraine from Ukrainian media). No user was exposed to the same news text attributed to different sources. To isolate the effects of the source country from the particular publication label, we chose to present source attribution in a generalized way: each news item was marked as coming “from Russian media,” “from Ukrainian media” or “from Kazakhstani media.”

Experimental Design and Procedure

We employed a 2x2x2 experimental design where each user received eight news items randomly retrieved from one of our subsets with varying veracity (true/false), narrative type (dominant/alternative) and randomly assigned news source origin (user’s country/the country covered in the news). Each respondent received all eight combinations of news features. News storage, random retrieval, and presentation, as well as survey presentation and collection of responses was administered through our software developed specially for this research.

Since a significant share of exposure to political news occurs on social networking platforms (Newman et al., 2021), we recruited participants via ad managing systems of the two social networking websites most popular in the post-Soviet region, Facebook and VK. In 2020 VK, despite its ban in Ukraine in 2017, was still relatively widely used there via VPNs, especially in the East, while Facebook was not yet declared an extremist organization and was not yet blocked in Russia. After clicking on our ad, users were redirected to our research software. We created two nearly identical user interfaces: a standalone website for Facebook users and a mobile app for VK users, with additional functionality used for our larger study. We informed the participants that the test measured their susceptibility to fake news. The entire research design was approved by the Ethical committee of (IRB of HSE University) and described in more detail in Vziatysheva et al. (2021).

The first page of our experimental interface contained a short task description, the link to the “About” page, and a consent checkbox. Next, users were shown nine consecutive screens with one news item on each screen including a distractor news item; our application drew the news items randomly from one of our 24-item sets so as to form individual subsets of eight news corresponding to our 2X2X2 factorial design. Participants were asked to evaluate each news item’s credibility on a six-point Likert scale ranging from “True” to “Fake.” Following the task, participants answered 21 questions relevant to the larger study. The questionnaire concluded with exhibiting the user’s accuracy score, a humorous one-liner summarizing their ability to tell real news from fake news, and an invitation to see the correct answers.

Independent Variables

In this paper we use several sets of questions from the larger study: 1) three demographic features (age, gender, and level of education); 2) variables registering whether users believed that they had seen the news and whether they checked the veracity of the news before they rated its credibility; 3) two questions that reflected the two most relevant dimensions of individual political views, as outlined in the Conceptual framework section: level of government support (“To which extent do you generally approve of the policies of your country’s leadership?” 1 – entirely disapprove, 5 – entirely approve) and the perceived quality of the political relations between the user country and the country covered in the news (“How can you describe the relations of your country with [COUNTRY_NAME]?” 1 – very hostile, 5 – very peaceful) (later referred to as perceived relations). Participants could abstain from responding to these two questions.

Outcome Measure

Message credibility was measured on a scale from 1 to 6 where 1 corresponded to the response “fake,” and 6 – to “true.” It should be noted that credibility is not equivalent to the correctness of the response, which is a different variable (1 – correct answer, 0 – otherwise) used in a separate study.

Participant Recruitment and Sampling

We recruited study participants using targeted advertisements on both social platforms, VK and Facebook, between March and July 2020. The subsamples were constructed to represent the audiences of the respective platforms in each country in terms of age, gender, and province. From each platform, five subsamples were collected: (1) Kazakhstani (KZ) users reading Russian news (RU) in the Russian language (RU) – further termed KZ-RU-RU; (2) Russian users shown news about Kazakhstan in the Russian language (RU-KZ-RU); (3) Russian users shown Ukrainian news about Ukraine in the Russian language (RU-UA-RU); (4) Ukrainian users shown Russian news in the Russian language interface (UA-RU-RU); (5) Ukrainian users shown Russian news in the Ukrainian language interface (UA-RU-UA). A detailed account of recruitment procedures, targeting, and sample balancing is presented in Appendix C. Key demographic characteristics of the final sample (N=8,559) can be found in Table 1.

Table 1. Sample statistics (top) and means (M) and standard deviations (SD) of credibility ratings (bottom).

	KZ-RU-RU		RU-KZ-RU		RU-UA-RU		UA-RU-RU		UA-RU-UA
N of users	1817		1836		2050		1731		1125
Gender (N)	f:906	m:911	f:851	m:985	f:942	m:1108	f:749	m:982	f:476	m:649
Support rating (%)	89	79	90	77	91	75	91	80	89	77
Perceived relation rating (%)	93	93	87	75	85	73	93	88	94	89
	M	SD	M	SD	M	SD	M	SD	M	SD
Age	35.429	10.409	38.236	11.206	38.734	11.395	37.375	11.791	40.297	11.8
Support	2.724	1.34	2.621	1.287	2.552	1.271	2.49	1.236	2.306	1.216
Perceived relation	4.337	0.817	4.027	0.819	2.612	1.172	1.727	0.991	1.517	0.792
N news seen	0.098	0.17	0.04	0.109	0.079	0.165	0.087	0.181	0.078	0.168
N news checked	0.03	0.116	0.012	0.07	0.017	0.087	0.027	0.117	0.014	0.081
Truth status
• true	3.473	2.09	3.782	2.015	3.929	1.991	3.777	2.071	3.922	2.069
• fake	3.05	2.051	3.116	2.009	3.056	1.993	3.124	2.073	3.26	2.119
Narrative
• dominant	3.344	2.082	3.64	2.051	3.739	2.032	3.774	2.074	4.086	2.032
• alternative	3.179	2.077	3.258	2.009	3.246	2.017	3.126	2.071	3.096	2.091
Source of news
• country of user	3.24	2.078	3.451	2.036	3.471	2.042	3.419	2.099	3.588	2.126
• country of news	3.283	2.084	3.447	2.043	3.514	2.037	3.481	2.095	3.594	2.115

Note. KZ = Kazakhstan, RU = Russia, UA = Ukraine; groups are coded by [user country – news country – news language; UNL]; KZ-RU- RU means Kazakhstani users reading Russian news in the Russian language. Support rating (%): Percentage of users who provided a rating of government support. For number of participants in different education and age groups see, Figure 1B and Supplement Figure S1. Users rated eight of 24 news items constructed for their UNL group; one each in the Narrative (2) x Type of news (2) x Source of news (2) conditions.

Figure 1. Moderating effects of narrative of users’ perceived relation of user and news country for contrasts (A) between countries not in conflict (KZ, RU) and countries in conflict (RU, UA), (B) two countries in conflict (UA vs RU), and (C) Ukrainian users choosing to read news in Ukrainian vs. choosing to read news in Russian. Error bars of means and shaded bands of smooths are 95% confidence intervals.

Data Analysis

We employed a linear mixed model (LMM) for simultaneously testing the following effects: (a) experimental effects of narrative, truth status of news, and source of news; (b) quasi-experimental effects relating to user country/language of news, age, gender, education, government support, perceived relations between the user country and the country covered in the news; and (c) the effects of binary variables indicating whether a user withheld a rating of government support, of perceived relations, and whether a news item was rated as seen before or as checked before the credibility rating. Effects related to expected differences between the five samples were specified with four orthogonal contrasts.

Specifically, with the first contrast UNL1, we compared users who saw the news about a country in a neutral relationship with their own country and users who saw news from a country in conflict with their own country (UNL1: KZ-RU-RU and RU-KZ-RU vs. RU-RU-UA, UA-RU-RU, and UA-RU-UA). With the second contrast UNL2, we compared KZ-RU-RU vs. RU-KZ-RU. With the third contrast UNL3, we compared RU-RU-UA vs. UA-RU-RU and UA-RU-UA. And with the fourth contrast UNL4, UA-RU-RU vs UA-RU-UA. The UNL acronyms stand for [U]ser country, [N]ews country, and [L]anguage of news, respectively. These four contrasts were allowed to interact with other experimental and quasi-experimental covariates. Further details about the specification of contrasts and interaction terms included in the model as well as the specification of the fixed effects and assessments of the complexity of the random-effect structure of the LMM are described in Appendices D and E; detailed results are documented in Supplement B, Tables S2 to S5.

Results

Overall, we analyzed 68,472 ratings of 96 news items given by 8,559 users from the three countries. In the upper part of Table 1, we summarize basic statistics about the number of users and the percentage of users providing a rating on the question of government support and the perceived relations with the news country. These results are separated by gender and reported for the five subsamples. The results indicate that male participants were more likely not to answer the question on government support than female participants in each of the five subsamples (overall: 22% of male non-respondents compared to 10% of female non-respondents). We observed a similar pattern for ratings of perceived relations (male: 17%; female: 10%).

The means and standard deviations for age, ratings of support of the government, perceived relations, and the number of news rated as seen and checked are reported in the middle block of Table 1. Users who had withheld a rating of government support or perceived relations were assigned a neutral response. Finally, we summarize the means and standard deviations for the main effects of narrative, news veracity, and source of news on credibility ratings in the bottom part of Table 1. In each of the five subsamples of users, news items representing a dominant narrative were rated as more credible than those affiliated with an alternative narrative. Also as expected, true news items were reliably perceived as more credible than fabricated ones.

Table 2 lists the significant (i.e., z-value > 2.0) fixed-effect terms of the LMM. The complete list of model parameters is provided in Supplement-B Table S2 for variance components and correlation parameters and Table S3 for fixed-effect estimates and test statistics. Corresponding fixed-effect estimates and test statistics are also provided for less complex LMM versions in Supplement-B Tables S4 and S5. The significance of fixed-effect terms did not depend on the complexity of the random-effect structures shown: all three LMMs yielded the same profile of significant fixed effects. Table 2 also provides references to figures and hypotheses for the various interaction terms. The main effects and interactions related to demographic covariates of gender, education, and age are documented in Appendix F.

Table 2. Fixed-effect estimates and standard errors (SE) with significant z-values (> 2.0) of LMM.

Parameter	Estimate	SE	z- value	Figures	Test
Grand Mean	3.874	0.053	72.87
Truth	0.171	0.048	3.57
Narrative	0.257	0.047	5.51		H1
Source	−0.017	0.007	−2.34		RQ2
Age	0.003	0.0007	4.21
Edu_low	0.04	0.015	2.66
Seen	0.469	0.023	20.79
Checked	0.068	0.033	2.07
UNL1 x Narrative	1.296	0.558	2.32		H2
UNL3 x	−0.120	0.042	−2.85
Support
Narrative x Support-NR	−0.061	0.023	−2.63
Narrative x Support	0.054	0.008	6.83
Narrative x Age	0.0028	0.0007	3.98	A2-A	demographic
Truth x Age	0.0048	0.0007	7.11	A2-B	RQ1; dmgr.
UNL2 x Age	−0.0132	0.0023	−5.79	A2-C	demographic
UNL1 x Seen	−0.919	0.27	−3.4
UNL2 x Seen	0.2	0.068	2.97
Truth x Gender	−0.056	0.012	−4.85	A1-A	RQ1; dmgr.
Truth x Edu_high	0.077	0.014	5.38	A1-B	RQ1; dmgr.
UNL1 x Truth x Source	0.248	0.087	2.85		RQ2
UNL1 x Truth x Narrative	−1.819	0.566	−3.22	2A	H2
UNL2 x Truth x Narrative	−0.382	0.126	−3.03	2B	H2
UNL2 x Narrative x Support2	0.043	0.018	2.4	3A	H3
UNL3 x Narrative x Support	0.084	0.042	2.51	3B	H3
UNL1 x Narrative x Relation	−1.109	0.037	−7.4	1A	H1
UNL3 x Narrative x Relation	0.319	0.062	5.19	1B	H1
UNL4 x Narrative x Relation	0.094	0.033	2.85	1C	H1; H4

Note. Support-NR: Effect of not having provided a rating of support of government. UNL2 x Narrative x Support2: Significant after adding quadratic term for Support; see Supplement B, Table S6 for details. Figure: reference to figure visualizing the interaction. Test: reference to hypothesis number in text or test of demographic feature.

We now turn to our main hypotheses and research questions. The analysis revealed a significant main effect of narrative type. The credibility of news representing a dominant narrative was significantly higher than of those representing an alternative narrative across all subsamples (b = 0.257, SE = 0.05, z = 5.51, p < .01), thus lending support to Hypothesis 1.

Simultaneously, we have tested whether the narrative has an effect distinct from individual views on international relations. As a corollary of Hypothesis 1, we expected that credibility ratings would depend on the users’ perceived relations between their own country and the country of the news they rated but would not be sufficient to account for the effect of narrative. As expected, given the manifest conflict between Ukraine and Russia, the number of ratings associated with the perceived quality of the relations was highly dependent on the true state of affairs and yielded a highly unbalanced number of ratings across the scale of perceived relations. Therefore, our interpretation of interactions of narrative and perceived relations focuses on the side of the scale receiving the expected pattern.

Despite this complication, the results are very clear: the effect of narrative is much larger than the effect of perceived relations and exists independently of it. However, there is also evidence for the effect of perceived relations on credibility. Figure 1A shows that ratings of dominant and alternative news diverge with an increase in the positivity of the perceived relations between the countries that are not in conflict with each other; the ratings also converge with a decrease in the negativity of the perceived relations between the countries that are in conflict with each other (Relation x Narrative x UNL1: b = −1.11, SE = .04, z = −7.4, p < .01).

The “conflict” pattern is also shown by Ukrainian users reading Russian news (Figure 1B, right facet), but Russian users reading Ukrainian news tend to lean toward the opposite divergence (Figure 1B, left facet; Relation x Narrative x UNL3: b = .32, SE = .06, z = 5.19, p < .01).

Figure 1C, finally, shows that the “conflict” pattern is more pronounced for Ukrainian users who chose to read Russian news in Ukrainian than for those who chose to read them in Russian (Relation x Narrative x UNL4: b = .094, SE = .033, z = 2.85, p < .01). There was no significant interaction associated with withholding this rating.

A test of Hypothesis 2 is the contrast between respondents who saw the news about a neutral country (Russians about Kazakhstan and Kazakhstanis about Russia) and those exposed to the coverage of a nation with which their home country is in a state of confrontation (Ukrainians about Russia and Russians about Ukraine). It is coded as UNL1 in our analyses and summary tables.

There is a significant overall difference in the effect of narrative between respondents seeing news about neutral versus adversary countries, as conflict appears to be boosting the credibility of dominant narrative news and dampening the credibility of alternative news (UNL1 x Narrative; b = 1.30, SE = 0.56, z = 2.32, p < .01). Thus, the findings support Hypothesis 2.

However, as shown in Figure 1A, when looking at true and fake news separately, a different picture emerges. The relationship between narrative and conflict between countries varied significantly between true and fake news items (UNL1 x Truth x Narrative; b = −1.82, SE = 0.57, z = −3.22, p < .01). A post-hoc LMM confirmed that narrative had no significant interaction with the contrast between conflict and non-conflict countries for true news (p = .54; Figure 2A, left facet), that is the UNL1 x Narrative effect originates almost entirely from fake news ratings (Figure 2A; right facet). Additionally, our analysis revealed no significant effect of narrative on the credibility of fake news in the no-conflict pair – against the backdrop of a sizable difference between dominant and alternative-narrative fake news in the conflicting pair of countries, strongly diverging relative to the rating observed for no-conflict groups (Figure 2B, right facet).

Figure 2. Interactions involving manipulation of narrative with (A) conflict between countries, (B) no-conflict countries of users (KZ vs. RU) x veracity of news. Error bars are 95% confidence intervals.

Another significant but qualitatively different interaction involving narrative and truth status was obtained when comparing Russian and Kazakhstani respondents who saw the news about each other’s countries (Figure 2B; UNL2 x Truth x Narrative; b = −0.38, SE = 0.13, z = −3.03, p < .01). In this case, Russian respondents perceived true news about Kazakhstan as significantly more credible if they represented the dominant narrative and vice versa for the alternative narrative (Figure 2B, left facet). In a post-hoc LMM, the corresponding interaction was not significant for fake news (p = .17; Figure 2B, right facet).

Hypothesis 3 predicted that respondents’ support of their government would moderate the effect of the narrative on credibility for Russian respondents but not for their Ukrainian and Kazakhstani counterparts. We specified two UNL contrasts to test this hypothesis: UNL2, which pitted Russian users reading news about Kazakhstan against Kazakhstani users reading news about Russia (Figure 3A), and UNL3, comparing Russian users reading news about Ukraine with Ukrainian users reading news about Russia (Figure 3B).

Figure 3. Moderating effects of narrative by government support for (A) contrast between countries in conflict (UA vs RU) and (B) countries not in conflict (KZ vs RU). (C) Ukrainian users choosing to read the news in Ukrainian or Russian. Smooth lines are second-order polynomial fits Error bars of means and shaded bands of smooths are 95% confidence intervals.

As seen in Figure 3A, for the Russia-Kazakhstan contrast, credibility ratings increase with self-reported government support for the dominant narrative and decrease for the alternative one. Consistent with our expectations, dominant and alternative narratives diverge much more strongly for Russian than Kazakhstani respondents (Support x Narrative x UNL2: b = 0.043, SE = 0.018, z = 2.40, p < .05). While the patterns are somewhat different in the middle, the extremes of the government support scale behave remarkably similar. In both countries, those the least supportive of their government exhibited no significant difference in credibility assessments of dominant and alternative-narrative news, while those the most supportive reported significantly higher ratings for dominant-narrative news.

For the Russia-Ukraine contrast, again, the relationship between credibility and the users’ support of the government is stronger (i.e., more positive for the dominant narrative and more negative for the alternative one) for Russian than Ukrainian users (Support x Narrative x UNL3: b = 0.084, SE = 0.042, z = 2.51, p < .01). Although the Support x Narrative x UNL4 contrast was not significant (b = .029, SE = .023, z = 1.28, p = .2003), it is informative to split the right facet of Figure 3B by the two groups of Ukrainian users (Figure 3C). Indeed, in a post-hoc LMM testing this interaction as nested within each of the five groups, the interaction was significant for the four groups reading the news in the Russian language, but not significant for Ukrainian users who read the news in the Ukrainian language (b = .021, SE = .020, z = 1.08, p = .2783). As visible in the figure, the difference between reported credibility scores of dominant-narrative and alternative-narrative news reported by this unique group is the greatest among all other subsamples, and it does not vary across levels of government support. Thus, the effect predicted in H3 was found in a larger number of subsamples than expected.

As noted above, a sizable percentage of users (22% male, 10% female) withheld their rating of government support. This user characteristic interacted with the narrative. Specifically, the credibility of ratings was higher for alternative news (3.24 vs. 3.18) for users who withheld their support rating than those who provided it; this difference was not significant for news written from a dominant perspective (3.69 vs. 3.67); Narrative x Support-NR: b = −.06, SE = .02, z = −2.63, p < .01. The pattern is compatible with the interpretation that users who withheld a rating of government support are more similar to users who do not support their government.

In H4, for Ukrainian respondents, we expected the effect of narrative type on news item credibility to be higher if the respondent chooses to read the news in the Ukrainian rather than the Russian language. This interaction was not significant in the LMM; (b = .029, SE = .023, z = 1.28, p = .2003). However, the interaction was qualified by a further moderation of perceived relations, already shown in Figure 1C; b=.094, SE = .033, z = 2.85, p < .01. Clearly, the interaction is present when we restrict the observations to those Ukrainian users who perceived the relationship with Russia as bad (1) or very bad (2). This provides partial support to H4.

RQ1 is about the effect of news veracity. Its significant interactions with variables of theoretical interest have been covered while addressing our hypotheses. There were, however, also three unexpected interactions involving gender, education, and age documented in Appendix F. First, female users were better at discriminating true and fake news (i.e., female users gave lower credibility ratings to fake news and higher credibility ratings to true news than male users; b = −0.056, SE = 0.012, z = −4.85, p < .01; see Figure A1-A). Second, the difference between the perceived credibility of true and fake news was larger for users with higher education than for the overall average (b = 0.077, SE = 0.014, z = 5.38, p < .01; see Figure A1-B). Third, credibility ratings increased with age only for true, but not for fake news (b = 0.0048, SE = 0.0007, z = 7.11, p < .01; see Figure A2-B). These results deserve to be followed up.

Finally, RQ2 is about the effect of news sources. Here, our analysis revealed significantly higher credibility when the news was attributed to a source from the country covered in the news than to a source from the user country (b = −.017, SE = .007, z = −2.34, p < .01). This main effect, however, was qualified by an interaction of the source with truth and the contrast of conflict between countries (UNL1 x Truth x Source: b = .25, SE = .09, z = 2.85, p < .01). The primary source effect was for true news and no-conflict users. Although the size of the source-related main effect and the associated interaction is very small, they are meaningful because the source of news is an experimentally manipulated design factor, not an observed feature of the news. It reflects a small media-level effect that is not accounted for by individual differences included in the model. Indeed, these source-related effects were only significant after statistical adjustments for individual differences.

Discussion

In this paper, we experimentally confirmed the intuition that news consumers’ judgments about the credibility of foreign affairs news are affected by a system-level factor (narrative): news items consistent with the dominant interpretations are perceived as more credible than those challenging these dominant narratives. This finding supports our hypothesis (H1). Although the effect is partially moderated by individual perceptions of the relations with a respective country and by the level of individual government support, these two moderating covariates have no effects of their own, while the narrative type has a highly significant unique effect. More importantly, it is larger than the effect of the other two experimentally manipulated variables – news truth status and news sources type (the latter being nearly negligible). In other words, narrative type contributes to the perceived news credibility more than the actual veracity of news, and news presented from an alternative narrative is almost always perceived as less credible except for a small subgroup of users in the non-conflict pair of countries who hold highly atypical views about the relations between these countries.

The larger effect of narrative as compared to news veracity that we find is somewhat at odds with the findings of Erlich and Garner (2023) who showed that Ukrainian news consumers are good at detecting fakes identified in Russian media by independent fact-checkers, irrespective of the propagandistic strategies applied. The reason for the difference is that in our experiment fakes were experimentally constructed to be very difficult to discriminate from true news. Given the difficulty users had with fake-true news discrimination, the effects of the narrative manipulation as well as users’ support of their government became the driving force of credibility ratings. Thus, Erlich & Garner’s focus pitted real true new against real fake news. Our experiment maximized the chances to observe the effect of narrative and government support under friendly and hostile constellations of international relations on credibility ratings.

As outlined in the Hypotheses section, a possible mechanism behind the narrative effect may be the cognitive pressure resulting from the intensive prior exposure to the dominant narrative that suppresses confidence in the plausibility of alternatively framed events. Importantly, this mechanism is distinct from prior exposure to the coverage of the event itself. The fact that perceived familiarity with a news item has its own – and, in fact, stronger – effect on the credibility rating does not undermine the effect of the narrative. In terms of political communication theory, this finding provides evidence not only for the existence of country-level dominance in media narrations about foreign affairs (widely shown since Galtung & Ruge, 1965), but more importantly also for the existence of country-level echo chambers. This holds at least in the sphere of foreign affairs news, a closed system of representations, where rebuttal is hardly possible even in peaceful times, while at times of tension the polarization of such echo chambers might only be higher.

We, therefore, hypothesized that international conflict would amplify the effect of the dominant narrative for users in adversarial countries exposed to news about each other. Indeed, the expected effect was revealed, yet only for fabricated news. Our analysis demonstrated that when Russian and Kazakhstani users were shown fakes about each other, they reacted to them with the same level of trust in the news exhibiting both dominant and alternative narratives. However, when Russian and Ukrainian users were presented with fakes about each other, on average they were much more likely to trust fake news if it was aligned with the dominant narrative. Conversely, they would considerably rate down the credibility of alternative fake news. In the Russia-Ukraine pair, dominant narratives of a respective neighboring nation were fiercely disparaging and often invoked references to various facets of the conflict. Alternative narratives were largely detached from the animosity or could even present some of the neighbor’s stances and policies favorably.

Thus, our study provides suggestive evidence that in the presence of an international conflict people can be more susceptible to misinformation casting the adversary in a negative light and less susceptible to misinformation that is more favorable toward the adversary. This evidence is suggestive because in our study both conflict and non-conflict conditions are represented by only one pair of countries, and thus more research is needed. Furthermore, the revealed effect is likely to have different explanations in such different countries as Russia and Ukraine. While in the latter, given its decentralized media control, the most plausible explanation is the “rally-round-the-flag” phenomenon (Baker & Oneal, 2001; Oneal & Bryan, 1995), in Russia, more consolidated state control over news production may be an additional or even the only explanatory factor.

Additionally, the interaction of narrative and conflict, although significant, was further moderated by individual perceptions of bilateral relations between the countries in all subgroups except Russians seeing news about Ukraine. This group is the only one in which the distribution of opinions about international relations is not heavily skewed toward the definitions of these relations offered by the regional experts (who defined Russian-Kazakhstani relations as peaceful and Russian-Ukrainian as hostile). This suggests that trust in dominant and alternative narratives converges (or even flips) for those individuals whose views of international relations are marginalized by their minority status. That is, the vast majority of participants in all subsamples exhibited a much higher trust in the dominant narratives, and this is especially pronounced for Ukrainian users. This suggests that during conflicts the responsiveness to the dominant narrative might be related to the degree to which the population of a given country suffers from the conflict: in the Russia-Ukraine pair, the latter has been a more affected party.

Another theoretically important moderator of the narrative’s effect on the credibility of foreign coverage was the participant’s level of government support, which we expected to be important in contexts where the dominant narrative significantly overlapped with the one strategically projected by the authorities (Russia). We also expected it to be less pronounced or non-existent in systems where elites exerted less control over the dominant narrative (Ukraine) or largely refrained from pushing aggressive messaging (Kazakhstan). Our expectations were confirmed only partially. We discovered that lower government support decreased trust in the dominant narratives and increased trust in alternative narratives in all subgroups except Ukrainians receiving news in Ukrainian. Additionally, in this group, the overall difference in credibility between the dominant and alternative narratives was the largest of all subsamples and significantly larger than even among Ukrainians receiving news in Russian. This calls for an explanation different from the one we had hypothesized initially.

The stage of stimuli formation revealed that, in fact, the dominant narrative largely overlapped with the government positions in all three countries, including Russia and Kazakhstan, where it was likely to be top-down government-induced discourse, and in Ukraine where it was more likely to be a bottom-up consensus. Despite these differences, by default, in the foreign affairs domain, low awareness and motivation to learn more seem to push government supporters to simply convert their trust in government into trust in the dominant narrative. Audiences critical of their respective governments who constitute the majority in our sample remain much more immune to the effects of dominant narratives. This happens unless the audience embraces the visions expressed by the dominant narrative to the extent that these internalized visions outweigh people’s opinions about their government and persist irrespective of how critical individuals are of their country’s leadership. We believe that this is what occurs in the subsample of Ukrainians who chose Ukrainian as the language of news. Since the Ukrainian language is a proxy for greater political and cultural allegiance to the Ukrainian statehood and given that the dominant narrative about Russia in Ukraine is conspicuously negative, it is not surprising that this subgroup assigns the highest credibility to such narrative across all levels of government support. The results are also in line with Erlich and Garner (2023) who found that Ukrainian-speaking Ukrainians are less likely to believe disinformation from the Russian media than Russian-speaking Ukrainians.

Limitations

This study has several limitations, most notably related to some of our design choices. While we took all available measures to minimize researcher bias in the process of constructing fake news items, some variance in credibility ratings may be due to unique features of our construction procedure rather than respondents’ biases. Also, concerning the limited effect of the source of news, the unspecific labels (e.g., “reported by Russian sources”) may have fallen short of eliciting a strong user response. More realistic source manipulation might yield more pervasive evidence of this effect. Furthermore, our decision to demonstrate news items in Russian only to Kazakhstani users may have discouraged some Kazakh-speaking individuals from participating in the study. This may have disproportionately affected users from Southern Kazakhstan, where the Kazakh language is especially prominent.

Conclusion and Future Research

In this research we have proposed the distinction between dominant and alternative narratives in news coverage of international affairs that offers a useful tool for political communication scholars seeking to explain variation in the public’s aggregate news credibility judgments. This approach captures the influence of the uniformity of news coverage of foreign countries on the public’s propensity to trust certain types of messaging. Across three countries, dominant narratives are shown to have a stronger effect on perceived news credibility than actual news veracity. This effect is reinforced under international conflict conditions, and although it may be to a certain degree compensated for by disloyalty to the government, this disloyalty effect fades when the hostility toward the adversary country is strong. We have offered a nuanced and context-specific description of the role of narrative for international news credibility by revealing a complex interplay of both amplifying and compensating factors. We have contributed to the refinement of political communication theory understood as a system of testable statements and to the rigor of experimental methodology in the field.

However, future research is needed to clarify whether the effect of narrative is amplified by the presence of international conflict or by specific differences between the examined country pairs. Our intuition here is that, since the rally-around-the-flag effect has been widely documented across different contexts, the observed amplification should replicate for other conflicts, especially in countries experiencing external invasions to territory consensually perceived as their own. Further, it remains to be seen whether the distinction between dominant and alternative narratives generalizes to domestic news, which is considerably more fragmented in terms of both issue repertoires and user attitudes. Apparently, in some national systems it would be possible to identify and operationalize dominant narratives, but not in others. For example, the polarized and relatively symmetric political spectrum of the Unites States limits identification of dominant narratives in certain domestic issues. Future studies may probe the limits of the concept’s applicability beyond foreign affairs news.

Finally, credibility ratings represent a mixture of knowledge and response tendencies to say “yes” or “no.” In the current analyses, we used two variables (i.e., ratings of having seen the news and having checked the news) to control for this effect. Such responses, however, occurred very rarely (see, Table 1). An alternative approach is to use signal detection theory and obtain measures of sensitivity (i.e., the ability to discriminate between true and fake news) separated from response bias based on users’ credibility ratings

Acknowledgement

The order of authors is alphabetical. Supported by the Basic Research Program at the National Research University Higher School of Economics (HSE University) in the year 2021. The authors gratefully acknowledge many helpful comments during the review process. Data as well as R- and Julia-scripts used for analyses are provided in OSF repository https://osf.io/t54u9/.

Disclosure Statement

No potential conflict of interest was reported by the author(s).

Supplementary Material

Supplemental data for this article can be accessed on the publisher’s website at https://doi.org/10.1080/10584609.2023.2172492

Additional information

Notes on contributors

Kirill Bryanov

Kirill Bryanov (PhD, Louisiana State University) is an independent researcher affiliated with the Laboratory for Social and Cognitive Informatics at HSE University, Saint Petersburg, Russia. His research focuses on how emerging communication technologies shape people's perceptions of society and politics.

Reinhold Kliegl

Reinhold Kliegl (PhD, University of Colorado) is a Senior Professor of Psychology in the Department of Sports and Health Sciences at the University of Potsdam with a recent focus on multivariate statistics for social media research.

Olessia Koltsova

Olessia Koltsova is the director of Social and Cognitive Informatics Lab and an associate professor of sociology at Higher School of Economics, St.Petersburg, Russia.

Alex Miltsov

Alexander Porshnev is a Senior Researcher in the Laboratory for Social and Cognitive Informatics at HSE University, Saint Petersburg, Russia. His research is focused on human-computer interaction, the effects of digital technologies, and various aspects of data analysis in social media research.

Sergei Pashakhin

Alex Miltsov (PhD, McGill University) is an Associate Professor in the Department of Sociology at Bishop's University. His research investigates the socio-economic and cultural effects of digital technology use and media representation.

Alexander Porshnev

Sergei Pashakhin is a PhD student at the Chair for the Governance of Innovative and Complex Technological Systems at the University of Bamberg. Their study is focused on the interaction of people with information and technologies in social and political contexts.

Yadviga Sinyavskaya

Yadviga Sinyavskaya (PhD candidate, Higher School of Economics) is a junior research fellow in the Laboratory for Social and Cognitive Informatics at HSE University, Saint Petersburg, Russia. Her research expertise is grounded in the field of human-computer interaction, interpersonal communication, and online studies.

Maksim Terpilovskii

Maksim Terpilovskii (PhD,) is a senior research fellow in the Laboratory for Social and Cognitive Informatics at HSE University, Saint Petersburg, Russia. His research is focused on human-computer interaction, interpersonal communication, and text mining methods.

Victoria Vziatysheva

Victoria Vziatysheva (MA) is a doctoral student in media and communication studies, an independent scholar, and a former journalist.

Notes

1. Full lists of media sources are provided in Appendix A.

2. For further examples of coverage categorized as dominant or alternative within various media systems refer to Appendix B and Supplement A.

Appendix

A. Lists of media sources

Table

	Kazakhstani Sources	Ukrainian Sources	Russian Sources
1	Informburo	Segodnya	Komsomolskaya Pravda
2	Zakon	Novoe Vremya	Kommersant
3	INFORM KZ	Obozrevatel	Vesti
4	Delovoy Kazakhstan	24 TV	REGNUM
5	BaigeNews	TSN	TASS
6	TengriNews	POLITEKA	RIA
7	Forbes KZ	Korrespondent	Echo Moskvy
8	News Times KZ	NEWSFRONT	Gazeta.Ru
9	365 INFO	Novostnoy Front	REN-TV
10	Sputnik News KZ	Delovaya Stolitca	360°TV
11	KazTAG	GORDON	LIFE
12	NUR KZ	Liga	Izvestia
13	Kazakhstan Today	LIGA	Moskovskiy Komsomoletc
14	Caravan	STRANA UA	5 Kanal
15	TOTAL	ZIK	Lenta
16	Kazakhstanskaya Pravda	Ukrinform	RBK
17	HOLA News	Gazeta UA	NTV
18	Vlast	Glavred	Svobodnaya Pressa
19	Nasha Gazeta	Komsomolskaya Pravda UA	Interfax
20	Habar 24	Leviy Bereg	TJournal
21	WEBINFO	Dialog UA	1 Kanal
22	Azattiq Radiosi	InfoResist	Novaya Gazeta
23	Village Kazakhstan	NovorosInform	Meduza
24	Central Asia Monitor	UNIAN	EuroNews
25	KZ_EXPERT	Shariy NET	Vedomosti
26	Neonomad KZ	Fraza	Mediazona
27	BARIBAR	Ekonomucheskaya Pravda	ZNAK
28	KazakhSTAN 2.0	ELISE JOURNAL	Current Time
29	Novosti Zhetysu	5 Kanal UA	RT
30	Novaya Gazeta Kazakhstan	Espresso TV	AIF

B. Narrative Discovery: Examples

The topics most frequently covered in each country pair were markedly different, as was the tone of coverage. For example, in Kazakhstani media covering Russia and the relationship between the two countries, a discussion of the Eurasian Economic Union, where both Kazakhstan and Russia are founding members, was prominent. The dominant way of framing the state of this economic union was complimentary, stressing the successes in the two nations’ economic collaboration and the mutually beneficial character of the relationship, and emphasizing how the two nations’ leaders work productively together to craft shared economic policies. The alternative way of talking about the Eurasian Economic Union, occasionally observed in the output produced by Kazakhstani media, was to the effect that Russia benefited from the union, while Kazakhstan did not. Another topic where systematic differences in framing were observed was the discussion of the 2018 presidential election in Russia: While the dominant narrative maintained that Vladimir Putin’s victory was legitimate and indisputable, the alternatively narrated coverage of the issue questioned the election’s integrity and thus the legitimacy of the outcome. A largely symmetric coverage of the 2019 presidential election in Kazakhstan was revealed in mainstream Russian media. Overall, the tone of the news coverage on both sides was predominantly reserved, with factual messaging significantly outweighing overtly opinionated content.

The analysis of news coverage of Ukraine by Russian media and of Russia by Ukrainian media revealed much more politically and emotionally charged messaging on both sides, fueled by the ongoing tensions between the two nations. The Ukrainian dominant narrative revolved around the unlawfulness of the Russian takeover of the Crimean Peninsula (with almost every mention of the territory accompanied by adjectives such as “occupied” and “annexed”); criticism of the authoritarianism and overall backwardness of the Russian political system; justifying Ukraine’s movement away from Russia and toward closer integration with the European Union; Russian leadership’s hostile intentions to forcefully maintain its influence over Ukraine. Against this background, any messages that, short of being favorable of Russia, merely normalized it and lacked strong condemnation of its policies toward Ukraine were coded as representing the alternative narrative. The narrative that dominated Russian mainstream media painted Ukraine as a dysfunctional state lacking geopolitical agency and being manipulated by foreign powers into serving their anti-Russian agenda. Ukrainian leadership and political system in general were presented as dysfunctional, dominated by radical nationalists, and committed to oppressing citizens who were open about their pro-Russian sympathies and identity. In this picture, the Ukrainian economy was also represented as struggling, if not crumbling, allegedly illustrating the futility of the nation’s newly found pro-Western political orientation. The alternative narrative distinguishable in Russian media in many regards aligned with the major tenets of the Ukrainian dominant narrative in that it condemned the actions of the Russian government toward Ukraine and cast Ukraine’s pro-Western turn in a positive light as a movement toward a more democratic society. In sum, while not always antithetical, elements of dominant and alternative narratives often focused on different facets of the neighboring country’s representation.

C. User Samples, Targeting, and Balancing

The most popular social networking platform in both Russia and Kazakhstan is Vkontakte, or VK, currently owned and operated by Russian conglomerate Mail.ru Group. Initially widely popular in Ukraine as well, in 2017 it saw a blanket ban as part of the Ukrainian government’s sanctions against Russia. As a result, Facebook became Ukraine’s prime social media platform, although some Ukrainians continued accessing VK via VPN services. To account for these contextual differences, we targeted audiences of both networks in all three countries. Additionally, in Ukraine we launched two separate campaigns in Russian and Ukrainian, using two identical sets of news. The Ukrainian-language recruitment effort on VK was predictably unsuccessful.

Despite online social networks providing a useful tool for recruiting academic study respondents, such platforms are not representative of national populations. Yet, in line with our goal of studying social media users’ perception of news credibility, we took additional steps to construct samples representative of national audiences on Facebook and VK. Having obtained data on gender, age, and regional composition of these populations via advertisement managing systems of these platforms, we calculated age-gender quotas for each region in each country and targeted each of the demographic groups with separate tailored advertisements. In all, we ran ten advertising campaigns on both platforms yielding over six million impressions and almost 76,000 clicks.

After slightly oversampling each of the quotas, we balanced the sample with a custom algorithm that excluded excess respondents. We discarded incomplete questionnaires, those submitted by underage users, and those from outside of the focal countries. Finally, we removed users without responses to the government support question. As mentioned above, a week-long Ukrainian-language recruitment campaign failed to collect a reasonable number of participants, so we stopped this data collection effort and dropped the obtained questionnaires from further analyses.

The data were collected between March and July 2020. We then cleaned and analyzed the data between September and October 2020. Given that our data collection coincided with the COVID-19 pandemic, we assumed that many users might have expectations of seeing COVID-related news. Therefore, we added a distractor news item about COVID at the beginning of the task.

D. Contrasts Specification

Considering that Ukrainian users could choose to read news about Russia in either Russian or Ukrainian language, a total of four groups of news items were shown to five groups of users, referred to as the between-subject UNL factor with the five levels: (1) KZ-RU-RU, (2) RU-KZ-RU, (3) RU-UA-RU, (4) UA-RU-RU, and (5) UA-RU-UA. The first pair of letters codes the country of the User, the second the country covered in the News, and the third the Language in which the news was presented.

We specified four orthogonal contrasts that in interaction with the other factors afford direct tests of our hypotheses. Specifically, Hypothesis 2 (moderating role of conflict) requires a contrast of the mean of KZ-RU-RU and RU-KZ-RU with the mean of RU-UA-RU, UA-RU-RU, and UA-RU-UA (users in Russia and Kazakhstan seeing news about each other’s countries vs. users in Russia and Ukraine seeing news about each other’s countries). Tests of Hypothesis 3 (moderating role of government support) rely on separate contrasts between Russian users seeing news about Kazakhstan and Kazakhstani users seeing news about Russia, and between Russian users seeing news about Ukraine and Ukrainian users seeing news about Russia. Hypothesis 4 is tested by contrasting the two subsamples of Ukrainian users.

E. Statistical Analysis

Software

Statistical analysis (preprocessing of data, descriptive statistics, generation of figures) was carried out in the R environment of statistical computing (R Core Team, 2021) using the RStudio interactive development environment and the tidyverse collection of packages (Wickham et al., 2019); also the cowplot package (Wilke, 2019). Inferential statistics (i.e., linear mixed model estimation and its post-processing) was initially based on the lme4 (Bates et al., 2015b), broom.mixed (Bolker & Robinson, 2021), and sjPlot packages (Lüdecke, 2020), but final LMM selection and estimation was carried out with the speed of the MixedModels.jl package available in the Julia programming language (Bates et al., 2020). Details about model selection, goodness-of-fit statistics, and lists of parameter estimates for three candidate models are documented in Supplement B.

Fixed-effect specification

Statistical significance was assessed with a linear mixed model (LMM) with user (n=8,559) and item (n=96) specified as crossed random factors contributing 68,472 ratings (observations). Experimental effects associated with the design factors narrative, truth status, and source were estimated as differences from the Grand Mean (GM; i.e., they were coded as effect contrasts). For quasi-experimental effects associated with the UNL factor (i.e., the five groups of different users) four orthogonal contrasts were estimated: (1) the difference between users from countries that are in conflict (i.e., Russian users reading Ukrainian news and the two Ukrainian groups) and users from countries that are not in conflict (i.e., Russian users reading Kazakhstani news and Kazakhstani users), (2) the difference between users from non-conflict country groups (KZ-RU-RU vs. RU-KZ-RU), (3) the difference between users from countries that are in conflict (RU-UA-RU vs. UA-RU-RU, UA_RU-UA), and (4) the difference between the two Ukrainian groups (UA-RU-RU vs. UA-RU-UA). The user’s self-rated support of their government and their perceived relationships between the countries were included as continuous covariates (linear trend) centered at the neutral rating of a five-point Likert scale. As these two ratings were optional, we also included two dummy covariates coding users who opted out of the respective answer. Quasi-experimental factors of rating the news as “seen” or “checked” were included with effect contrasts. Finally, we included age as a continuous covariate (linear trend) centered at the median of 35 years and gender (male/female) as well as education with effect contrasts. With this specification, the LMM intercept estimated the GM of credibility.

A baseline LMM varying only GM included all third-order interactions between the following six variables in the theoretical focus: UNL, narrative, support, perceived relations, truth status, and source as well as main effects and a subset of simple interactions with the control variables news seen, checked, age, gender, and education. We dropped three of five education contrasts that were never significant, leaving contrasts testing effects of low and high levels of education, respectively. This procedure ensured that all theoretically relevant interactions could be tested and that important sources of variance relating to potentially confounding variables would not be overlooked.

Selection of random-effect structure for linear mixed model

Fixed-effect statistics based on an LMM estimating variance components (VCs) only for subject- and item-related GMs are possibly anti-conservative because within-subject and within-item-related effect VCs and correlation parameters (CPs) are not taken into consideration. Following recommendations by Bates et al. (2015a) and Matuschek et al. (2017), we dropped non-significant VCs of all eligible main effects while forcing all CPs to zero in a first step and extended this reduced LMM with all possible CPs in a second step. The resulting parsimonious LMM was supported by the data (i.e., not overparameterized). Both steps led to a highly significant improvement in the goodness of fit (see Supplement B, Table S1). Model selection was carried out without any knowledge of its impact on the significance of fixed effects, but these statistics were not critically affected by the complexity of the random-effect structure (see Supplement B, Tables S3 to S5 for fixed-effect estimates and test statistics of LMMs with complex, zero-correlation parameter, and varying-only-intercepts random-effect structures).

List of variables

Table

Short name	Description	Model level	Type	Mean	SD
Subj	Anonymized ID of study participant	Random	Numerical
Item	ID of a news item presented to a participant	Random	Numerical
age	Age of a participant	Covariate	Numerical	37.9	11.4
Edu	Level of education attained (six levels)	Covariate	Ordinal
Gen	Gender identity	Covariate	Binary
SNS	Online social platform (OSP) of recruitment (two levels)	Covariate	Binary
C_set	Indicator for an obs. in dimensions: Country, OSP, chosen language (six levels)	Covariate	Nominal
p_rel	“How can the relations between your country and [country condition] be characterized?”	Covariate	Numerical	2.95	1.58
r_na	Indicator of whether a respondent declined to answer about the relationships between countries (1 = declined)	Covariate	Binary
p_sup	“To which extent do you generally approve the policy of the government of your country?”	Covariate	Numerical	2.47	1.39
s_na	Indicator of whether a respondent declined to answer about support for government (1 = declined)	Covariate	Binary
p_hel	“Most of the time people try to be helpful”	Covariate	Numerical	3.03	1.37
p_tru	“The level of general trust to people”	Covariate	Numerical	2.29	1.29
p_use	“Most people would try to take advantage of you if they got a chance”	Covariate	Numerical	3.71	1.32
Believe	Did a participant believe a news item?	Covariate	Binary
See	Participant’s report on seeing a presented news item before	Covariate	Binary
Che	Participant’s report on fact checking a presented item	Covariate	Binary
UserCountry	Country of participant (three levels)	Covariate	Nominal
NewsCountry	The country in coverage of news items presented to a participant (three levels)	Covariate	Nominal
Language	The language a participant chose for the study (Ukrainian vs. Russian)	Covariate	Binary
UNL	Participant’s group in dimensions: their country, covered country, the chosen language	Design Variable	Nominal
Source	Presented source of an item: foreign or domestic outlet	Design Variable	Binary
Truth	Indicator whether an item is true of fabricated	Design Variable	Binary
Nrtv	Indicator of a narrative type in an item: dominant vs. alternative	Design Variable	Binary
answer	Participant’s judgment of item truthfulness	Dependent	Numerical	3.44	2.07
accuracy	Participant’s accuracy score for distinguishing true items from fabricated	Dependent	Numerical	5.08	1.55

F. Effects of Gender, Education, and Age

We would like to highlight significant effects related to three of respondents’ demographic characteristics: gender, education, and age. While the sample is not representative of the target nations’ general populations, the large sample size, and weighting procedures that we undertook prior to the analyses support the notion that demographic profiles of our respondents approximate those of VK and Facebook user populations in these countries. We can therefore draw valid inferences about the association between demographic features such as gender, education status, and age on the one hand and susceptibility to online misinformation in these groups on the other.

Gender. Female users rated true news as more, and fake news as less credible than male users (Figure A1-A). In other words, women, on average, were significantly better at distinguishing between true and fake news messages (b=−0.056, SE=0.012, z=−4.85, p < .01).

Education. The analysis yielded two main effects of education. Respondents with completed vocational and higher education reported higher credibility ratings than the Grand Mean b=0.040, SE=0.015, z=2.66, p < .01). In addition, the difference between perceived credibility of true and fake news (i.e., truth discernment) was larger for users with higher education than for the overall average (Figure A1-B; b=0.077, SE=0.014, z=5.38, p < .01). Note that a truth effect of similar magnitude was obtained for users with an academic degree but, due to the comparatively small number of users, this contrast was not significant.

Figure A1. Effects of gender (panel A) and education status (panel B) on the perceived credibility of real and fake news. Error bars are 95% confidence intervals.

Age. There was also a significant increase of credibility ratings with age for conditions with high (above 3.5) credibility means, but not for conditions when credibility was below this value. Specifically, there are three significant interactions of age with narrative (Figure A2-A; b=0.0028, SE=0.0007, z=3.98, p < .01), truth status of news (Figure A2-B; b=0.0048, SE=0.0007, z=7.11, p < .01), and the UNL2 contrast (Figure A2-C; b=−0.0132, SE=0.0023, z=−5.79, p < .01).

Figure A2. Interactions of age with narrative (panel A), truth status of news (panel B), and UNL2 contrast (panel C) on the perceived credibility of real and fake news. Error bars are 95% confidence intervals.