Document language moderates the text-belief consistency effect

ABSTRACT

When researching socioscientific topics, particularly on the Internet, readers face multiple texts that they must integrate into a coherent mental model. Previous research in monolingual settings has found that comprehension is biased toward readers’ prior beliefs (text-belief consistency effect). Considering that the Internet is multilingual, this experiment focused on the influence of document language on the text-belief consistency effect. Assuming differences in epistemic prestige among languages, we hypothesized that document language would act as a credibility cue and moderate the text-belief consistency effect. Seventy-four German university students read two conflicting texts either in their L2 English (higher epistemic prestige) or in their L1 German (lower epistemic prestige). Participants constructed stronger situation models for the belief-consistent text than for the belief-inconsistent text only when they read the texts in German. These results indicate that document language can act as a source characteristic that may reduce belief biases in comprehension.

Readers intending to learn more about socioscientific issues on the Internet often encounter multiple documents that may contain controversial information. When reading these documents, prior knowledge and beliefs about the researched topic are activated, which can lead to a biased representation of information: Readers tend to develop stronger situation models for belief-consistent information (text-belief consistency effect; Richter & Maier, 2017). Such text-belief consistency effects have been found in a number of studies with different text types and measures of comprehension and memory (e.g., Abendroth & Richter, 2020, 2023; Kobayashi, 2018; Van Strien et al., 2016; Wiley, 2005), however, only in monolingual contexts. Considering the multilingual character of the present-day information society, particularly the Internet (Lee, 2017), it is important to extend this research into bi- and multilingual settings. In a first study addressing this issue, Karimi and Richter (2023a) examined the text-belief consistency effect in a bilingual context (Persian versus English) and found document language to be a boundary condition of the effect. Readers who read the texts in their first language, Persian, showed a stronger text-belief consistency effect than readers who read the texts in their second language, English. Karimi and Richter attribute this pattern of effects to the higher epistemic status of English as the lingua franca of science. Our study followed up on these considerations by replicating the effect with a different topic and language pair (English versus German) and examined perceived document credibility as a potential mediating mechanism of the moderating role of document language.

Validation and belief biases in comprehension

Belief biases in comprehension can be explained by the process of validation. During text processing, readers construct a mental representation, known as a situation model (Van Dijk & Kintsch, 1983), about the state of affairs described in a text by integrating information from the text with prior knowledge and beliefs. They constantly update the situation model during reading when new information is presented by comparing this new information to the current situation model and accessible prior knowledge and beliefs in terms of consistency, which is called validation (O’Brien & Cook, 2016; Richter, 2015; Singer, 2013). Validation is a nonstrategic and automatic process that occurs without a specific reading goal or epistemic motivation. It is crucial for establishing a coherent situation model but may also lead to a biased representation of information if strong beliefs are involved in the validation process (Isberner & Richter, 2014). In the context of (multiple-) text comprehension, such belief biases are known as text-belief consistency effects (Richter & Maier, 2017, 2018): The situation models for texts that are consistent with readers’ prior beliefs are often found to be stronger than the situation models for texts presenting belief-inconsistent information. Ideally, readers should strive for an unbiased representation when researching about a specific controversy (Schroeder et al., 2008). Research has identified conditions that help readers to avoid belief biases, such as strong prior knowledge (Wiley, 2005), alternating presentation of belief-consistent and belief-inconsistent texts (Maier & Richter, 2013), adopting a belief-inconsistent reading perspective (Abendroth & Richter, 2023; McCrudden et al., 2017), an “evaluatist” profile of epistemological beliefs (Barzilai & Eshet-Alkalai, 2015; Karimi & Richter, 2021), increasing source awareness (Richter & Maier, 2017, 2018) or thinking dispositions (Karimi & Richter, 2023b). However, this research has been pursued in monolingual reading contexts only and has not considered the multilingual nature of current knowledge societies. In the following, we will elaborate on why the document language might be an important moderating condition for the text-belief consistency effect.

Importance of language as a source characteristic

The linguistic landscape of the Internet has changed from a medium that once was predominantly-English to a multilingual medium (Dor, 2004; Lee & Barton, 2012). Additionally, users have also become bi- or multilingual (Maher, 2017). Nowadays, users gather information online not just in their first language but also in English. Research results, in particular, are often published in English, as the lingua franca of science (Ammon, 2001; Hamel, 2007; Liu, 2017). De Swaan (2001) has described an epistemic hierarchy of languages with English being the “hyper-central” world language, followed by some “super-central” languages (e.g., German) and many “central” languages (e.g., Persian). We expect these differences in perceived epistemic prestige, especially between English and other languages, to influence text understanding as they might induce different levels of credibility and make language a source characteristic.

Source characteristics are content-external attributes of information documents, such as the author or the place of publication. Source characteristics can influence the perceived credibility of an information source and the extent to which information is processed and integrated into the situation model (Britt & Rouet, 2012; Wertgen & Richter, 2020). The Documents Model Framework (Britt et al., 2013; Perfetti et al., 1999) assumes that, when reading multiple documents with conflicting information, readers construct an intertext model, where text information is tied to the respective source information. The representation of sources and source characteristics allows readers to consider the trustworthiness of information, which affects how information is weighed in readers’ situation models when reading multiple texts (Bråten et al., 2016) and can benefit their learning outcomes (e.g., Anmarkrud et al., 2014; Bråten et al., 2009; Strømsø et al., 2010; Wiley et al., 2009). According to Wertgen et al. (2021), whether source information is considered in the process of routine validation depends on the degree of (im-)plausibility. For information that is consistent with a reader’s knowledge or beliefs, source credibility should have little influence on validation. However, for text information that is inconsistent with readers’ knowledge and beliefs, source credibility is considered in the validation process and can affect both implicit and explicit plausibility judgments—unless the text information is clearly false or unacceptable in the light of readers’ beliefs. In support of this assumption, Wertgen et al. (2021) showed that reading times and plausibility judgments for information embedded in little stories varied as a function of knowledge consistency. However, highly credible sources (i.e., protagonists with high expertise) mitigated the disruptive effects of partially implausible sentences on reading times and the negative effects of these sentences on plausibility judgments. Against this background, we assumed that document language should modulate the validation of belief-inconsistent information and the ensuing effects on comprehension. More specifically, the higher credibility tagged to information presented in English (compared to the same information in German) should make it more likely for belief-inconsistent information to be further engaged with and included into the reader’s situation model.

We assume document language to function as a source characteristic that readers use for evaluations of credibility. In contrast to other source characteristics that readers tend to disregard even though they are relevant (Britt & Aglinskas, 2002; Kobayashi, 2014), document language is a source characteristic that is difficult to ignore.

Document language and multiple-text comprehension

Only one study has examined the role of document language for the text-belief consistency effect. Karimi and Richter (2023a) had 87 Persian students read two texts about the global spread of English in their L1 (Persian) and/or their L2 (English). They found a text-belief consistency effect that was smaller for the group that read both texts in English compared to the group that read both texts in Persian. For the group that read one text in each language, the effect was stronger when the belief-consistent text was presented in Persian and weaker when it was presented in English. Karimi and Richter attribute these findings to the higher epistemic prestige of the English language. They reason that belief-inconsistent texts weigh more strongly when presented in a language with higher epistemic prestige (English) compared to the same information presented in a language with lower epistemic status (e.g., Persian).

Rationale of the present study

The findings by Karimi and Richter (2023a) provide initial evidence for the moderating role of document language on the text-belief consistency effect. One aim of the present study was to replicate these results conceptually with a different topic and language combination to shed light on the generalizability of the assumption and improve on limitations mentioned by Karimi and Richter (2023a). In their study, texts about the global spread of English were used, which might have affected participants’ awareness of the language in which the texts were written. The present experiment was based on texts about a socioscientific issue unrelated to language (utility versus risk of genetically modified food). Moreover, we chose a topic where both stances (pro and contra) had supporters and opposers in our sample. Finally, we improved the comparability between the texts in L1 and L2, as our English texts were accurate translations of their German text counterparts. Additionally, we examined the language combination English versus German. The latter is considered a central language (De Swaan, 2001) but still lower in epistemic status than English.

A second aim of this study was to extend the previous results by testing the assumption that text credibility is the mechanism that mediates the moderating influence of document language on the text-belief consistency effect.

We expected an overall text-belief consistency effect across both languages (Hypothesis 1). The effect should be moderated by the document language in such a way that the effect is larger in the group that read the texts in German, the language with lower epistemic status, compared to the group that read the texts in English, the language with higher epistemic status (Hypothesis 2). Finally, we expected the perceived document credibility to mediate this interaction effect (Hypothesis 3).

Method

Results for the evaluation study of the stimulus material can be found in the Online Supplement (https://osf.io/yxpwn/?view_only=b86ede973d89436db530ad5f7dc78b0c, S2).

Participants

Seventy-four participants (Mean age = 25.80 years, SD = 9.32) took part in the study. Fifty-eight were female, 15 were male, and one participant did not specify a gender. The majority were students (85.14%) and German native speakers (90.54%). They received either course credit or monetary compensation (10 Euros) for participation. Five participants were excluded from the analyses: one whose L1 was English, one who showed an unrealistically fast reading time of less than 1 minute (average reading time: M = 5.54 min, SD = 1.76), and three who reported conspicuously extreme attitudes and were clearly identifiable as outliers (deviating more than 2.5 SDs from the sample mean and lower than 1.5 times the interquartile range beyond the first quartile, Online Supplement S7) in the prior belief questionnaires.

Experimental texts

The material of the study consisted of two texts, one highlighting the advantages (pro-stance text) and one highlighting the dangers (contra-stance text) of genetically modified crops, each in a German and an English version. The issue was selected because a pilot study revealed it to be controversial among German students (Online Supplement S1). All texts were identical in structure and contained four arguments. The texts summarized research results about genetically modified crops and were roughly equivalent in length and reading difficulty across text stances and languages. The texts were between 907 and 1,015 words long (M = 996.50, SD = 41.34) with a mean difficulty (Flesch, 1948, German adaptation by Amstad, 1978) of 35.23 (SD = 3.54), which can be considered “difficult” to read but appropriate for an academic setting.

Procedure and measures

Participants’ prior beliefs were assessed online via 10 statements presented in German about genetic modification (Likert scale from 1, disagree completely, to 7, agree completely) about 1 week (M = 6.78 days, SD = 2.28) prior to the experiment to avoid carryover effects. The prior belief scores were centered to the scale mean, scores ranging from −3 to 3; zero indicated neutrality, positive values indicated that participants supported genetic modification, and negative values indicated that they opposed it.

The experiment proper took place in the laboratory, where participants read the pro-stance text and the contra-stance text in random order in either English or German on a laptop. Each text was presented in four sections, one per argument. After one text was read, participants were presented with a recognition task, modified after Schmalhofer and Glavanov (1986), to measure situation-model strength. It consisted of 24 sentences, eight per item type (paraphrases, inferences, distractors). Participants were asked to indicate whether each item occurred in the text, and situation-model strength was computed based on the proportion of yes responses to inference items (hits) to yes responses to distractor items (false alarms). The scale for situation-model strength covers a range of −3.29 (all items incorrect) to 3.29 (all items correct). Additionally, participants were presented with a credibility questionnaire consisting of eight adjective items from Appelman and Sundar (2016) and Karlsson et al. (2014) and were asked to rate to which extent the text (or the source) could be described by each adjective on a scale of 1 (not at all) to 7 (completely). The credibility score consisted of the mean of all items. The same procedure was repeated for the second text. Afterward, participants’ proficiency in English was assessed with the LexTALE (Lemhöfer & Broersma, 2012) to control for participants’ reading skills in their L2-English as differences might occur in the comprehension processes of readers when they read in their L1 versus their L2 (Horiba, 1996). Finally, participants were asked two questions to rate the relevancy of English and German in science on a Likert scale of 1 (not at all) to 7 (really relevant). These questions were added to check whether our theoretical assumptions about the difference in epistemic prestige of the two languages would indeed hold in this sample. More-detailed description of all measures is presented in the Online Supplement (S3).

Design

The design was a 2 (text-stance: pro-text versus contra-text, varied within-subjects) x 2 (document language: English versus German, varied between-subjects) design. The dependent variable was the situation-model strength of each text; document credibility was assessed as potential mediator. The language condition was assigned at random.

Results

All hypotheses were tested with the Type-I Error probability set to .05 (one-tailed for directional hypotheses unless stated otherwise). All analyses were conducted in R (version 4.2.2., R CoreTeam, 2022). Descriptive statistics and intercorrelations between all variables are reported in the Online Supplement (S4).

Perceived relevancy of English and German for science communication

We used the relevance ratings to test whether participants in our sample attributed a higher epistemic status to English compared to German. Descriptive data revealed greater relevancy for the English (M = 6.55, SD = 0.74) than for the German language (M = 4.45, SD = 1.72), and a Wilcoxon sign-ranked test confirmed the ratings for the English language to be significantly higher (Mdn = 7) than those for the German language (Mdn = 5), z = −5.82, p < .001. Participants’ ratings supported the assumption that English is perceived to be more relevant than German for dissemination of scientific knowledge.

Influence of document language on the text-belief consistency effect

We estimated a linear mixed model (LMM) with the lme4 package (Version 1.1.-33, Bates et al., 2015) to test whether a text-belief consistency effect occurred (Hypothesis 1) and whether this effect would be moderated by document language (Hypothesis 2). The LMM included a random intercept of participants and fixed effects of text-stance (pro = 1, contra = −1), prior beliefs as a metric variable (with 0 representing a neutral stance), language (English = 1, German = −1), and their interactions. Preliminary models that included English proficiency as a metric predictor did not differ substantially in their results. Therefore, this control variable was not included in the analysis. Estimates and significance tests of the fixed effects are presented in Table 1.

Table 1. Estimated coefficients, standard errors, degrees of freedom, and t values for the linear mixed model of the situation-model strength.

	Est.	SE	df	t	p
(Intercept)	1.86	0.14	105.64	13.34	<.001
Text-Stance^a	−0.08	0.14	65.00	−0.57	.285
Prior Beliefs	−0.20	0.19	105.64	−1.07	.144
Language^b	−0.53	0.20	105.64	−2.65	.005
Text-Stance*Prior Beliefs	0.43	0.19	65.00	2.25	.014
Prior Beliefs*Language	0.55	0.25	105.64	2.19	.015
Text-Stance*Language	0.20	0.20	65.00	1.00	.161
Text-Stance*Prior Beliefs*Language	−0.45	0.26	65.00	−1.77	.041

N = 69.

^aContrast coded (pro-texts = 1, contra-texts = −1).

^bContrast coded (English = 1, German = −1).

The interaction of prior beliefs and text-stance was significant: b = 0.43; t(65) = 2.25; p = .014. Participants with high prior beliefs had stronger situation models for the text that was in favor of genetically modified crops; whereas, participants with low prior beliefs had stronger situation models for the text that was against genetic modification (Figure 1). Thus, a text-belief consistency effect occurred, supporting Hypothesis 1.

Figure 1. Situation-model strength by text-stance and prior beliefs.

Gray areas represent 95% CI. Positive prior belief scores indicate belief consistency with the pro-text, negative scores indicating belief consistency with the contra-text. The scale for situation-model strength covers a range of −3.29 (all items answered incorrectly) to 3.29 (all items answered correctly).

To test whether document language moderates the size of this effect, we examined the interaction between prior beliefs, text-stance, and language. The interaction turned out to be significant: b = −0.45; t(65) = −1.77; p = .041. In line with Hypothesis 2, the text-belief consistency effect was present in the group who read both texts in German, b = 0.43; t(34) = 2.36; p = .012, but not in the group that read both texts in English, b = −0.03; t(31) = −0.15; p = .440 (Figure 2). Point estimates and associated significance tests for the differences between pro- and contra-stance texts at different levels of prior beliefs are reported in the Online Supplement (S5).

Figure 2. Situation-model strength by text-stance, prior beliefs, and document language.

Gray areas represent 95% CI. Positive prior belief scores indicate belief-consistency with the pro-text, negative scores indicating belief-consistency with the contra-text. The scale for situation-model strength covers a range of −3.29 (all items answered incorrectly) to 3.29 (all items answered correctly).

Analyses of the mediating role of credibility

Finally, we tested the hypothesized mediating role of credibility (Hypothesis 3) with the stepwise method proposed by Baron and Kenny (1986). We expected (1) an interaction effect of document language and the text-belief consistency effect on credibility, (2) a main effect of credibility on situation-model strength, and (3) a reduced influence of the interaction of document language on the text-belief consistency effect after adding credibility as a predictor to the model described in the previous section.

Regarding Step 1, we found a significant interaction between the text-stance and prior beliefs: b = 0.47; t(65) = 1.71; p = .046. Participants with positive prior beliefs rated the pro-stance texts as more credible; whereas, participants with negative prior beliefs rated the contra-stance texts as more credible. This pattern matches the text-belief consistency effect. However, unexpectedly, document language did not moderate the interaction: b = −0.10; t(65) = −0.27; p = .394. Contrary to our expectations, situation-model strength and credibility barely correlated (pro-texts: r = −.13, p = .283; contra-texts: r = −.02, p = .861). Therefore, Hypothesis 3 was not supported by the data and we skipped the further steps of the mediation analysis.

Discussion

The current experiment investigated the influence of document language on comprehension biases in reading. Results show that the document language moderates the text-belief consistency effect: The effect was present in the group that read texts in their L1 (German) but absent in the group that read the same texts in their L2 (English). This interaction was not mediated by perceived document credibility.

The bias in situation-model strength in favor of the belief-consistent text in the group that read the texts in German is consistent with previous findings (e.g., Abendroth & Richter, 2020, 2023; Maier & Richter, 2013) and can be attributed to the readers automatically checking the validity of text information against prior beliefs (Isberner & Richter, 2014). The absence of the effect in the group that read the texts in English is in line with the findings of Karimi and Richter (2023a). Belief-inconsistent information seems to be better integrated into the situation model when presented in an epistemically-more-prestigious language (English). Unexpectedly, however, participants’ beliefs were uniformly and positively associated with comprehension in the English language group: Participants’ situation models were stronger the more they endorsed the position that the advantages of genetically modified crops outweigh their potential dangers. We can only speculate about the causes that underly this positive relationship. One possible explanation is based on the observation that people with high knowledge about science and also people with better education tend to see more benefits of genetically modified foods (Pew Research Center, 2016). If a similar relationship holds within our sample, the higher prior knowledge of participants who see more benefits of genetically modified crops might account for the overall better situation model in the group who read the English texts. In that case, a third variable that is extraneous to validation and belief effects would have caused the deviation from the overall absent or weaker effects of prior beliefs that we would have expected in the group of participants who read the English texts. This unexpected result notwithstanding, the interaction of beliefs and text-stance indicating a text-belief consistency effect occurred in the German text group but not in the English text group, which we view as the central finding of this study.

The assumption that readers consider document language as a cue to assess the credibility of text information received no support. A possible explanation could be that document language is not an explicit source characteristic (as is, for example, author credentials) but guides readers at a more subconscious level and, therefore, was not captured by our explicit questionnaires. According to Wertgen and Richter (2020), source credibility can already modulate routine validation processes. Following up on their results, Wertgen et al. (2021) showed that the disruptive effects of somewhat implausible information on reading times and on plausibility judgments can be mitigated by high credibility sources. If document language functions as a credibility cue on the level of routine validation processes (Step 1 of the 2-Step Model of Validation, Richter & Maier, 2017), it might well be that readers do not become aware of it and do not consider document language in their explicit credibility judgments.

This study compared two languages in an imbalanced design, which raises the question whether the difference in processing between the two language groups might be due to differences in processing L1 versus L2 or processing German versus English. The language with the higher epistemic status, English, was always the L2 of the participants; whereas, the language with the lower epistemic status, German, was always the L1. Processing information in L2 uses more cognitive resources (Morishima, 2013), is less efficient (Horiba, 2000), and might be more difficult for integrating belief-inconsistent information into the situation model (Romero-Rivas et al., 2017). This is supported by the fact that a main effect of language is evident both on the level of the situation model and on the text base level (see Online Supplement S8 for full model on text base level): German texts are better understood in our German sample than English texts. We cannot rule out that differences in processing between L1 and L2 interfere with the effect found in this study. Nevertheless, there are aspects of the data speaking against this possibility. If the differences in reading comprehension in the L1 and the L2 groups are due to more-demanding cognitive processes in L2 reading, there should be a difference in comprehension in the group reading the English texts depending on language proficiency (Horiba & Fukaya, 2015). Additional analyses including English language proficiency as a moderator in the group that read both texts in their L2 English (see Online Supplement S9 and S10) revealed no evidence for a moderating role of this variable: No text-belief consistency effect occurred on any level of English language proficiency. Additionally, in the study by Karimi and Richter (2023a), the expected interaction of language with the text-belief consistency effect occurred even though participants did not differ in their level of comprehension between the L1 (Persian) and the L2 (English). These findings in a different language setting (English and Persian) also suggest that the focal effect in the present study cannot be attributed to differences in processing the English compared with the German language. Nevertheless, further research is needed to shed more light on how different processing in L1 and L2, as well as in specific languages, interacts with the text-belief consistency effect when reading multiple controversial documents. The issues arising from the imbalanced design of the study should also be addressed in future research. Since the experimental manipulation of the prestige of a language is a difficult endeavor, future studies should instead make additional comparisons of different language combinations in samples with varying L1 and L2 affiliations.

To conclude, the present study found the document language to reduce or even eliminate biases toward belief-consistent information in reading comprehension. The experiment provides initial evidence for this phenomenon being independent of the scientific field of the controversy and the compared languages. A difference in epistemic prestige between languages is just one possible explanation for the moderating effect of document language, which necessitates further research. In further examinations of this hypothesis, it might be more appropriate to use implicit rather than explicit assessments of credibility to find out whether the document language operates as an implicit credibility cue that affects validation processes. Additionally, future studies should explore additional different socioscientific issues and language comparisons to obtain more information about the generalizability of the effect. Considering that much reading about socioscientific issues takes place online in a multilingual setting, these findings encourage future research into the role of document language as a source characteristic.

Acknowledgments

We thank Anna Voigt and Hanna Weigand for their help in collecting the data.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

The experimental texts, data files, and R-scripts for the full analyses are available at https://osf.io/yxpwn/?view_only=b86ede973d89436db530ad5f7dc78b0c.

Additional information

Funding

This research was funded by the German Research Foundation (Deutsche Forschungsgemeinschaft, DFG, grant RI 1100/15-1).