Reading Comprehension Skills and Prior Topic Knowledge Serve as Resources When Adolescents Justify the Credibility of Multiple Online Texts

Abstract

This study sought to understand how well students (n = 274; M_age = 12.45) were able to identify the author, the main claim, and the supporting evidence (identification performance) and to justify the author’s expertise, the author’s benevolence, and the quality of the evidence (justification performance) while reading multiple online texts. The study also examined the contribution of prior topic knowledge and basic reading skills (word recognition and reading comprehension) to students’ identification and justification performance. Students read two more and two less credible online texts about sugar effects on health. After reading each text, they responded to multiple-choice items that measured the identification and justification performance. Justifying credibility seemed more challenging for students than identifying the claim, evidence, and author. Word recognition and reading comprehension were statistically significant predictors of identification performance, whereas prior knowledge and reading comprehension were statistically significant predictors of justification performance. The findings offer new insights into the relationship between basic reading skills and credibility evaluation that can inform both theory and instruction.

Introduction

Navigating the current online textual landscape, characterized by the rapid spread of misinformation, algorithmic bias, and persuasive argumentation (Kozyreva, Lewandowsky, & Hertwig, 2020), requires critical reading. To be able to read online texts critically, readers need to identify elements crucial to credibility (viz., the main argument and the source of the text) and evaluate the quality of the argument and the trustworthiness of the source (e.g., Barzilai, Thomm, & Shlomi-Elooz, 2020). To make accurate evaluative judgments of text credibility, readers need to understand why specific types of evidence can or cannot support authors’ claims well and why the source can or cannot be regarded as trustworthy. In other words, critical readers can justify their evaluative judgments.

While several previous studies have examined how adolescents justify their overall text evaluations (e.g., Forzani, Corrigan, & Kiili, 2022; Potocki et al., 2020), we aimed to gain a more nuanced understanding of adolescents’ justifications by examining how they justify the source’s expertise and benevolence as well as the quality of evidence supporting the text’s main claim. Further, we examined two essential cognitive resources - prior topic knowledge and basic reading skills - that may contribute to adolescents’ identification (i.e., identification of source, claim, and evidence) and justification performance (e.g., Forzani, 2018; Kanniainen, Kiili, Tolvanen, Aro, & Leppänen, 2019). Regarding reading comprehension, we also sought to gain a nuanced understanding by examining what kinds of reading comprehension processes might contribute to adolescents’ identification and justification performance. These examinations of adolescents’ justification skills and underlying cognitive resources can contribute to building a highly needed theoretical understanding of credibility evaluation, which, in turn, may help educators design instruction to support students’ critical reading skills in online contexts.

Content-Based and Source-Based Evaluation of Information

Several models or frameworks suggest that readers can engage in content- and source-based evaluation to determine whether they can trust offline or online information (Barzilai et al., 2020; Forzani et al., 2022; Stadtler & Bromme, 2014). For example, the content-source-integration model by Stadtler and Bromme (2014) suggests that readers can make first- or second-hand credibility judgments when encountering conflicting information in multiple texts. When making first-hand judgments, readers compare text content to their prior knowledge to determine whether the claims stated in the text are valid, whereas second-hand judgments are based on source information (e.g., source’s expertise, intentions) that readers evaluate to determine whether they can trust the source of the text. Readers tend to rely on their prior knowledge or beliefs when the text content is more familiar to them, whereas in evaluating more unfamiliar topics, readers put more emphasis on source information (e.g., Bråten, McCrudden, Stang Lund, Brante, & Strømsø, 2018; McCrudden, Stenseth, Bråten, & Strømsø, 2016).

Barzilai et al. (2020) have elaborated on the notion of first- and second-hand evaluation. Their bi-directional model of first and second-hand evaluation strategies specifies three types of first-hand evaluation strategies: knowledge-based validation, discourse-based evaluation, and corroboration. Consistent with the content-source-integration model (Stadtler & Bromme, 2014), knowledge-based validation refers to comparing text content to one’s prior knowledge and beliefs, which is usually a routine part of comprehension (Richter, Münchow, & Abendroth, 2020). In discourse-based evaluation, readers draw on discourse features to determine how well knowledge is justified or communicated. For example, readers can evaluate the strength of the argument (McCrudden & Barns, 2016; Münchow, Tiffin-Richards, Fleischmann, Pieschl, & Richter, 2023) or the quality of evidence (List, 2023; List, Du, & Lyu, 2021). Finally, readers can engage in corroboration (see Wineburg, 1991) to examine whether other texts verify or contradict the content of the currently processed text.

Drawing on Stadtler and Bromme (2014), Barzilai et al. (2020) also specified different kinds of second-hand evaluation strategies targeting source trustworthiness. By using sourcing strategies, readers can evaluate, for example, the expertise or benevolence of the source of the text (e.g., Thomm & Bromme, 2016). Importantly, Barzilai et al. (2020) highlighted that first- and second-hand evaluation strategies are not used in isolation; instead, readers can employ them reciprocally, meaning that evaluation of information can affect evaluation of sources, and vice versa.

Students’ Content-Based and Source-Based Justifications

Identifying components of arguments (i.e., claim, reasons, and evidence) and sources of information is a prerequisite for content- and source-based evaluation. Readers may struggle to identify the main claim in naturalistic texts (Diakidoy, Ioannou, & Christodoulou, 2017; Larson, Britt, & Larson, 2004), although the simplicity and explicitness of the structure of the argument can facilitate the identification of the argument (Christodoulou & Diakidoy, 2020). And, while identifying source features in short, linear texts may be straightforward, it seems to depend on students’ decoding skills (Potocki et al., 2020). In online contexts, identifying the author might be more challenging, however, as the location of author information may vary, or students may confuse the author with other source information, such as affiliation (Coiro et al., 2015). For example, 17% of middle school students (n = 773) have been shown to have difficulties identifying the author of a web page (Coiro et al., 2015). Failing to identify the main claim, the supporting reasons, and the author may impede students in using sources to qualify the validity of the claims (Perfetti, Rouet, & Britt, 1999) and evaluate the quality of presented arguments (cf. Christodoulou & Diakidoy, 2020).

Students’ skills in using content- and source-based evaluation strategies have been studied by employing think-aloud methodology (Barzilai, Tzadok, & Eshet-Alkalai, 2015; Mason, Boldrin, & Ariasi, 2010; McGrew, 2021), or by asking students to justify their credibility evaluations or credibility rankings in writing (Braasch, Bråten, Strømsø, & Anmarkrud, 2014; Coiro et al., 2015; Potocki et al., 2020). Think-aloud studies have suggested that students’ spontaneous credibility evaluation during online inquiry is rather limited (Barzilai & Zohar, 2012; Kammerer, Gottschling, & Bråten, 2021) or superficial, even when students have been prompted to evaluate online information (McGrew, 2021). For example, Barzilai and Zohar (2012) examined sixth graders’ (n = 38) evaluation strategies by means of think-alouds and retrospective interviews. In that study, students completed two tasks: a search task on the open internet and a reading task, including three pre-selected, conflicting websites. When reading websites, students typically considered the content or the form of the website. However, source trustworthiness was only evaluated in 39% of the websites, and attention to scientific evidence was even more scarce.

Similar results have been found among high school students when examining students’ evaluation of online information about social and political topics (McGrew, 2021). McGrew (2021) found that students (n = 18) often based their credibility judgments on topical relevance or superficial features (e.g., date, absence of author, or hyperlinks) but hardly consider the trustworthiness of the sources. In addition, when students checked whether the author provided some evidence, they did not evaluate its quality.

Although think-aloud methodology can reveal students’ epistemic thinking in action (Barzilai & Zohar, 2012; Ferguson, Bråten, & Strømsø, 2012), students may vary in how comfortable and capable they are thinking aloud (Pressley & Afflerbach, 1995). Because of the resource-intensive nature of this methodology, the sample sizes are usually relatively small compared to many studies analyzing students’ written justifications. Moreover, similar patterns have been found across studies on written justifications and think-aloud studies: many students lack efficient evaluation strategies (e.g., Breakstone et al., 2021; Coiro et al., 2015; Potocki et al., 2020). For example, Potocki et al. (2020), who examined evaluation skills among 5^th-, 7^th-, and 9^th-graders and undergraduates (n = 245), found that especially in the lower grades, students emphasized more content than source information when justifying their credibility evaluations. Similarly, Coiro et al. (2015) found that many middle school students’ (n = 773) justifications for the author’s expertise, the author’s point of view, and the overall credibility of online texts were quite often unacceptable or superficial.

It is worth noticing that previous studies also have found considerable inter-individual differences among students, indicating that some students are able to justify the credibility across various perspectives and also display thorough reasoning while evaluating the texts (e.g., Barzilai & Zohar, 2012; Forzani et al., 2022; Hämäläinen, Kiili, Räikkönen, & Marttunen, 2021). Forzani et al. (2022) highlighted qualitative differences between better and poorer evaluators (n = 410) by examining 7th graders’ responses when asked to justify the credibility of one web page during the completion of an online inquiry task. Interestingly, better evaluators were more likely to refer to the source (89% of the better evaluators) than poorer evaluators (14% of poorer evaluators). A similar portion of better (13% of better evaluators) and poorer evaluators (16% of poorer evaluators) evaluated the content beyond just referring to it. Notably, better evaluators seemed to display more sophisticated reasoning in their responses.

Although students’ written justifications seem to be a fruitful method for capturing variations in students’ justification skills, there may be several factors that can affect the results. First, insufficient writing skills may hinder some students from expressing their thinking (cf. McCarthy et al., 2022). Accordingly, tasks requiring written explanations seem to be more challenging than tasks in which students do not need to explain their reasoning (Sparks, van Rijn, & Deane, 2021). Second, explaining one’s reasoning is a challenging literacy task that requires behavioral and cognitive engagement, and not all students are willing to invest their full effort (Goldhammer et al., 2014; List & Alexander, 2018). To overcome these potential issues, this study assessed students’ justification ability with multiple-choice items. We focused on understanding how students justify both the credibility of the source (expertise and benevolence) and the content (i.e., the quality of evidence).

Prior Knowledge and Basic Reading Skills in Justifying the Credibility

Previous research suggests that several cognitive skills or resources contribute to students’ credibility evaluation (see Anmarkrud, Bråten, Florit, & Mason, 2022 for a systematic review of individual differences in sourcing), including prior topic knowledge and reading skills. Prior topic knowledge and reading skills can be considered closely related, as the construction of a coherent text representation presumably relies heavily on readers’ prior knowledge (Kintsch, 1988). Kintsch’s (1988, 1998; see also McNamara & Magliano, 2009) construction–integration model of text comprehension describes how comprehension occurs through an interaction between textual content and readers’ prior knowledge. The construction and integration processes result in representations that are called a textbase and a situational model, respectively. The textbase refers to the underlying meaning of explicit information in the text, whereas the situation model refers to the meaning of the text that readers construct by drawing inferences that go beyond the concepts explicitly stated in the text and integrating the textbase with their prior knowledge.

Consequently, prior knowledge also has been considered an important resource for credibility evaluation (Brand-Gruwel, Kammerer, van Meeuwen, & van Gog, 2017; Lucassen, Muilwijk, Noordzij, & Schraagen, 2013). Previous studies have found that students’ prior topic or domain knowledge is positively related to their credibility evaluations (e.g., Braasch et al., 2014; Forzani, 2018; Kammerer et al., 2021). However, not all studies have observed statistically significant associations between prior knowledge and credibility evaluation (Hämäläinen et al., 2021; Mason et al., 2018).

Further, the role of basic reading skills in credibility evaluation has drawn researchers’ attention. Previous studies have examined associations between credibility evaluation and one or two aspects of basic reading skills. For example, Dyoniziak, Potocki, and Rouet (2023) did not find any association between word reading fluency and eight graders’ (n = 90) credibility evaluation performance measured with items that required discrimination between the most and least credible websites and justification of source credibility (see also, Braasch et al., 2014). However, a positive association between word reading fluency and students’ ability to infer sources’ intentions was observed. Hämäläinen et al. (2021) found a positive association between reading fluency and the ability to justify credibility among upper secondary school students. Also, learners with reading difficulties seem to struggle more with credibility evaluation than do other learners (Kanniainen et al., 2022).

Whereas the results are somewhat mixed with respect to lower-level reading skills, prior research suggests that reading comprehension is essential to students’ credibility evaluation, at least for younger students. A large-scale study (n = 1431 seventh graders) by Forzani (2018) showed that reading comprehension contributed to students’ credibility evaluation after controlling for prior knowledge and gender. Notably, when Kanniainen et al. (2019) examined the role of both reading fluency (measured with three tests) and reading comprehension in sixth graders’ (n = 426) evaluation of online texts, reading comprehension was the only statistically significant predictor after considering for students’ prior knowledge, gender, spelling, and nonverbal reasoning. These findings highlight that although lower-level reading skills may be a prerequisite for skillful credibility evaluation of texts, comprehension skills probably matter more.

However, reading comprehension is a complex skill that includes different types of comprehension processes, such as understanding words, locating information, identifying main ideas and authors’ purposes, making inferences at various levels, integration, interpretation, and evaluation, not all of which are equally demanding for the reader (Afflerbach, Cho, & Kim, 2015). For example, different comprehension processes are required based on the difficulty of the text and the task (Afflerbach et al., 2015) or depending on the text genre (Gersten, Fuchs, Williams, & Baker, 2001). Reading comprehension is typically assessed with batteries that include a combination of test items aiming at covering many different comprehension processes (Afflerbach, 2017), with higher performance also reflecting student’s ability to use higher-level comprehension processes in reading (cf. OECD, 2019).

While much is known about the different comprehension processes that underlie reading comprehension as a composite (e.g., Ahmed et al., 2016; LARCC & Logan, 2017), less is known about associations between the different processes. To the best our knowledge, no study has examined associations between different comprehension processes required in multiple-choice reading comprehension tests and justification skills involved in credibility evaluation. Therefore, in addition to examining how word recognition and reading comprehension were associated with adolescents’ identification (i.e., identification of the author, claim, and evidence) and justification performance, we also explored how different reading comprehension items might contribute to their identification and justification performance.

Research Questions

The research questions of this study were as follows:

1. How well can students identify the author, main claim, and supporting evidence in online texts?

2. How well can students justify the author’s expertise, the author’s benevolence, and the quality of evidence?

3. To what extent do students’ prior topic knowledge and basic reading skills predict their identification and justification performance?

4. Are different types of reading comprehension items differentially associated with students’ identification and justification performance after controlling for prior knowledge and word recognition?

Methods

Participants

Participants were 274 Finnish sixth graders (M_age = 12.45; SD = 0.32); 53.3%, 44.9% boys, 2.6% responded other or did not respond. Most students spoke Finnish (91.2%) at home. Altogether, 498 guardians (272 as Guardian 1 and 226 as Guardian 2) reported their educational background when completing the consent form, with 59% having a higher education degree (i.e., from a university or a university of applied sciences). In the Finnish population, 44% of citizens have a higher education degree, indicating that guardians with a higher education degree were somewhat over-represented in our sample (Official Statistics of Finland, 2020). The ethical statement was received by the Ethics Committee of the Tampere Region (No. 38/2019).

Critical Online Reading Task

Students completed a critical online reading task created with the Critical Online Reading Research Environment (Kiili, Räikkönen, Bråten, Strømsø, & Hagerman, 2023). In this task, students read four researcher-designed online texts about sugar’s effects on children’s hyperactivity (Texts A and B) and memory (Texts C and D) (see Figure 1). Two of the texts (B and C) were more credible, presenting information in accordance with current scientific knowledge (i.e., main claim and supporting evidence). In contrast, two less credible texts (A and D) were not in accordance with current scientific knowledge. In addition, the texts were manipulated regarding the author’s expertise, the author’s benevolence, and the publication venue.

Figure 1. Online texts.

When completing the task, students received guidance from fact-checker Max, who gave them the task assignment and provided some feedback. Max asked students to read one text at a time. After reading each text, students were asked to respond to identification, evaluation, and justification items that appeared one by one on the right side of the text. Identification items asked students to identify the author, the main claim, and the supporting evidence among three alternatives for each. After students had locked in their responses, Max provided feedback by revealing the correct answer. These identification items and the related feedback ensured that all students would evaluate the correct author and evidence. We formed a sum score for identification performance (maximum score: 12 p.). The reliability estimate (McDonald’s omega ω) for students’ identification performance score was .55.

Evaluation items asked students to evaluate the author’s expertise, the author’s benevolence, and the quality of evidence, respectively, on a 6-point scale. After each evaluation, students were asked to justify their evaluation (justification items) by selecting one response among four alternatives (see Tables 2–4 for justification alternatives). In formulating of the justification alternatives, we utilized open-ended responses from upper secondary school students (Kiili et al., 2022) and sixth graders (a pilot study). We created a sum score based on 12 justification items (three for each text) to form a justification performance variable. The maximum score was 12 points, and the reliability estimate (McDonald’s omega ω) for students’ justification performance score was .66.

The order of the items is shown in Figure 2 using the newspaper article (Text A) as an example. Students were randomly assigned to read the texts in two different orders. Group 1 read the texts in the order Text A, Text B, Text C, and Text D (coded as 0), whereas Group 2 read the texts in the order Text B, Text A, Text D, and Text C (coded as 1).

Figure 2. Flow of the task with example items from the newspaper article.

Other Measures

Word recognition

Word recognition was measured with a time-limited word chain test (Holopainen, Kairaluoma, Nevala, Ahonen, & Aro, 2004). The test consisted of 25 word chains each of which contained four words written without any space between the words. Students had 90 s to identify as many words as possible by drawing a vertical line between the words. Students’ score was the number of correctly identified words (max score 100). According to the test manual, test–retest reliability varies between .70 and .84.

Reading comprehension

Reading comprehension was measured with a test included in a Finnish standardized reading test battery (Lerkkanen, Eklund, Löytynoja, Aro, & Poikkeus, 2018). Students read a 1.5 page long informational text about cave paintings and responded to 12 multiple-choice items. All except one item had four alternatives with one correct response. The one item asked students to order eight statements in the order they appeared in the text. Students had the text available when responding to the items. They had 30 min to complete the task and could use the recess if needed. The items required students to either locate information from the text, make inferences, or evaluate text content. The maximum score was 12 points, and the reliability estimate (McDonald’s omega ω) for the reading comprehension score was .65.

Prior topic knowledge

The prior knowledge test included 12 true–false items about sugar (sample item: There are different types of sugar) and its effects (sample item: A specific sugar is essential for the functioning of the brain). The items were reviewed by a former health science teacher and a medical expert. The reviews resulted in small modifications of some expressions and the replacement of two slightly ambiguous items that were further checked by the medical expert. The revised items were used to test upper secondary school students’ prior knowledge (Kiili et al., 2022), and the test was also piloted with sixth graders. Based on the piloting, three items were modified for the sixth graders. The test-retest reliability was examined with 58 readers and the score was .64.

Procedure

We accessed the school classes through Microsoft Teams. In the first lesson, students completed the basic reading tasks, and in the second lesson, the credibility evaluation task. The researcher briefly introduced the tasks and showed a video with more detailed instructions for each task. Using the videos enabled us to keep the instructions constant across the classrooms. Teachers handled classroom management and communicated with a researcher via chat or microphone. Students completed the credibility evaluation task on a computer. They accessed the task with a code and completed it at their own pace (mean time on task was 20 min and 11 s, SD = 5:31).

Statistical Analyses

The statistical analyses were conducted by using SPSS 27. We conducted four sequential regression analyses (Tabachnick & Fidell, 2013). Regression Analyses 1 and 2 examined how prior topic knowledge, word recognition, and reading comprehension were associated with students’ identification and justification performance, respectively. As we were interested in the contribution of each variable, we conducted each analysis in three steps. In Step 1, we entered prior topic knowledge because it can be considered a fundamental component in reading comprehension (Kintsch, 1998). We entered word recognition (Step 2) before reading comprehension (Step 3) because the former can be considered lower-level reading skills relative to the latter. This order allowed us to examine the predictability of reading comprehension for students’ performance over and above their prior topic knowledge and word recognition.

Regression Analyses 3 and 4 were conducted to clarify which of the reading comprehension test items were associated with students’ identification and justification performances. The two first steps in the regression analysis were the same as in Regression Analyses 1 and 2. In Step 3, we entered all 12 reading comprehension items separately.

There were some missing data for all variables. The missing data (see Table 5) was due to absence from school (16 students were absent from one of the two lessons), an incomplete reading comprehension task (10 students), a misunderstanding of the word recognition task instruction (1 student), and a technical issue (1 student). Missing data were excluded pairwise. Little’s (1988) test indicated that missingness was completely random (χ²(13) = 6.27, p = .936).

Results

Identification of the Author, Main Claim, and Evidence

As shown in Table 1, most students were able to identify the author. However, some students confounded the author with the publisher or platform. Notably, 17.7% mixed up the author (journalist) with a person interviewed in the text. Further, students performed slightly better in identifying the main claim (varied from 71% to 86%) than evidence (53% to 92%). Students struggled to understand that the journalist relied on an expert statement in the newspaper article (53%) and that the CEO relied on a customer survey (56%).

Table 1. Responses to the items requiring identification of the author from three options.

Author (publication venue)	Author (correct)	Venue	A person mentioned in the text	Total
CEO (Company website)	90.57	5.28	4.15	100
Mother (Blog platform)	80.00	11.70	8.30	100
Researcher (Research Center website)	75.85	21.13	3.02	100
Journalist (Newspaper website)	66.79	15.47	17.74	100

Justifications for the Author’s Expertise

Justifying the author’s expertise required students to pay attention to source information and consider its relevance to the topic of the online text (see Table 2). Students struggled most in justifying the journalist’s expertise. Only about one third (30.6%) of the students justified the journalists’ expertise with his profession, whereas one third paid attention to the amount of information in the text and one fourth mixed up the author with an embedded source (i.e., a doctor interviewed in the article). In contrast, more than half of the students (57.0%) justified the researchers’ expertise with relevant source information. When justifying the mother’s and the CEO’s expertise, 54.3% of the students recognized that the mother was not a health expert, whereas the corresponding percentage for the CEO was only 33.6%. Many students justified the CEO’s expertise with his work related to sugar products.

Table 2. Responses to the justification items concerning the author’s expertise.

Justification alternative	f	%
Researcher in health sciences (Research Center website)
She is a researcher in health issues. (correct)	151	56.98
She seems to know a lot about sugar.	65	24.52
She does not tell about her own experiences about sugar effects.	29	10.94
She has written an informational text about sugar.	20	7.54
Total		100
Journalist (Newspaper website)
His text contains a lot of information.	90	33.96
He is a journalist who writes about health. (correct)	81	30.57
He is a doctor [the journalist interviewed a doctor].	64	24.15
He is unsure because he asks "true or a myth?" [in title].	30	11.32
Total		100
Mother (Blog platform)
She is not an expert in health issues. (correct)	144	54.34
She is a mother of three children.	57	21.51
She tells a lot about sugar effects.	47	17.74
She has written about the issue on the internet.	17	6.42
Total		100
CEO (Company website)
His work relates to sugar products.	116	43.77
He is not a health professional. (correct)	89	33.58
He can tell about sugar effects.	35	13.21
He is a leader of a company.	25	9.43
Total		100

Justifications for the Author’s Benevolence

Justifying the author’s benevolence required students to consider the author’s intentions. As shown in Table 3, students performed best in justifying the mother’s persuasive intention. Although the commercial online text included marketing statements, only 63.0% of the students justified their benevolence judgments with commercial intentions. Notably, more than half of the students struggled to understand that the researcher and the journalist were in professions that ideally require the sharing of unbiased information.

Table 3. Responses to the justification items concerning the author’s benevolence.

Justification alternative	f	%
Researcher in health sciences (Research Center website)
Her job is to share unbiased knowledge. (correct)	156	58.87
She only puts forward her own opinion.	64	24.15
She tries to change school meals.	29	10.94
She brags about her accurate knowledge.	16	6.04
Total	265	100
Journalist (Newspaper website)
His job is to tell accurate information. (correct)	153	57.74
He tries to show that parents are wrong.	75	28.30
He has written about the issue on the internet.	24	9.06
He just tries to get a lot of readers.	13	4.91
Total	265	100
Mother (Blog platform)
She tries to get parents to change their behavior. (correct)	215	81.13
She has written a text on the internet.	24	9.06
It is her obligation to provide accurate information.	21	7.92
She has a lot of readers.	5	1.89
Total	265	100
CEO (Company website)
He tries to increase the sales of his products. (correct)	167	63.02
He tries to inform his customers.	54	20.38
He shows that he is good at his job.	23	8.68
He writes on his companys website.	21	7.92
Total	265	100

Justifications for the Quality of Evidence

Justifying the quality of evidence required students to consider how well the author supports the main claim with evidence (Table 4). It was challenging for students to understand that the mother could not verify the causal claim with her own observation: only one-fourth of the students questioned the validity of her observation as evidence. In contrast, almost half of the students (47.9%) questioned the validity of the company’s customer survey as evidence. Further, half of the students understood the value of research studies as evidence. It is worth noticing that some students seemed to think that evidence should be produced by the author. That is, 22.7% selected the option that questioned the quality of the evidence because the interviewed expert did not rely on her own experiences. Additionally, 17.4% chose the option that questioned the research evidence because the author did not conduct the reported research.

Table 4. Responses to the justification items concerning the quality of evidence.

Justification alternative	f	%
Research evidence (Research Center website)
She uses two published studies. (Correct)	135	50.94
Research has been carried out by foreign researchers.	53	20.00
She has not conducted the studies that she reports.	46	17.36
The research results are contradictory.	31	11.70
Total	265	100
Expert statement (Newspaper website)
The expert is a doctor. (correct)	101	38.26
The expert’s report seems accurate.	74	28.03
The expert does not share her own experiences.	60	22.73
The expert says that children eat too much sweets.	29	10.98
Total	264	100
Own experience (Blog platform)
She saw the change with his own eyes.	120	45.28
She relies only on one observation. (correct)	67	25.28
She showed that only sugar could have caused the hyperactivity.	61	23.02
She describes the events precisely.	17	6.42
Total	265	100
Customer survey (Company website)
One could have selected the results that are favorable to the company. (correct)	127	47.92
Several students have responded to the survey.	79	29.81
Respondents have eaten candy before the exam.	40	15.09
The main results of the survey have been published on the company’s website.	19	7.17
Total	265	100

Contributions of Prior Knowledge and Basic Reading Skills to Students’ Identification and Justification Performance

Table 5 presents descriptive statistics for the variables included in regression analyses 1 and 2. Table 6 presents the results of the regression analysis predicting students’ identification performance (identification of the author, claim, and evidence). In Step 1, prior topic knowledge explained 5% of the variance in identification performance, F(1, 245) = 13.69, p < .001. In Step 2, the model, including prior knowledge and word recognition, explained 16% of the variance, F(2, 244) = 23.82, p < .001. After including reading comprehension in the final model in Step 3, 34% of the variance in identification performance were explained, F(3, 243) = 41.10, p < .001. Only word recognition and reading comprehension were statistically significant predictors in the final model. Thus, the better students’ basic reading skills were, the better was also their identification performance.

Table 5. Descriptive statistics and inter-correlations for variables included in the regression analyses.

Variable (observed range)	1	2	3	4	5
1. Identification performance (2–12)		.51***	.23***	.37***	.56***
2. Justification performance (1–12)			.31***	.31***	.57***
3. Prior topic knowledge (4–12)				.19**	.27***
4. Word recognition (4–40)					.42***
5. Reading comprehension (0–12)
N	264	264	265	266	257
Mean	9.22	5.98	8.01	18.84	6.76
Standard deviation	2.01	2.66	1.75	6.52	2.45
Skewness	−0.94	0.29	0.00	0.32	−0.09
Kurtosis	−0.85	−0.64	−0.34	0.46	−0.31

Note.**p < .01 ***p < .001

Table 6. Results of sequential regression analysis for variables predicting students’ identification (author, claim, and evidence) performance (N = 246).

Variable	β	p	R^2	ΔR^2
Step 1			.05***
Prior topic knowledge	.23	<.001
Step 2			.16***	.11***
Prior topic knowledge	.17	.006
Word recognition	.34	<.001
Step 3			.34***	.17***
Prior topic knowledge	.07	.189
Word recognition	.16	.006
Reading comprehension	.47	<.001

Note. ***p < .001

Table 7 presents the results of the regression analysis predicting students’ justification performance. In Step 1, prior topic knowledge explained 9% of the variance in students’ justification performance, F(1, 245) = 25.13, p < .001. After adding word recognition to the model (Step 2), the model explained 16% of the variance, F(2, 244) = 22.62, p < .001, with both prior topic knowledge and word recognition being statistically significant predictors.When reading comprehension was included in the equation in Step 3, the final model explained 35% of the variance in justification performance, F(3, 243) = 43.75, p < .001. Only prior knowledge and reading comprehension were statistically significant predictors in the final model. This means that the more prior knowledge students had about the text topic and the better reading comprehenders they were, the better they also wherein justifying the credibility.

Table 7. Results of sequential regression analysis for variables predicting students’ justification performance (N = 246).

Variable	β	p	R^2	ΔR^2
Step 1			.09***
Prior topic knowledge	.31	<.001
Step 2			.16***	.06***
Prior topic knowledge	.26	<.001
Word recognition	.26	<.001
Step 3			.35***	.19***
Prior topic knowledge	.16	.004
Word recognition	.07	.228
Reading comprehension	.50	<.001

Note. ***p < .001

Associations between Specific Reading Comprehension Items and Identification and Justification Performance

All reading comprehension items were positively correlated with identification and justification performance (see Appendix for Spearman correlations). We ran two additional regression analyses to investigate which kinds of comprehension items predicted students’ identification and justification performance. In these analyses, Steps 1 and 2 were the same as in the regression analyses presented above (see Tables 6 and 7). However, instead of using the comprehension sum score in Step 3, we entered all twelve reading comprehension items into the model.

Regarding the identification performance, the final model was statistically significant, F(14, 233) = 10.16, p < .001, explaining 38% of the variance. Four reading comprehension items (Item 1: β = .20, p < .001; Item 4: β = .13, p = .024; Item 9: β = .16, p = .004; Item 11: β = .15, p = .014) were statistically significant predictors. These items asked students to identify the text genre (1) and the purpose of the text (9), and to locate a detail presented in one particular sentence (11), in addition to the eight sentences that students were asked to number in the order they appeared in the text (4). Items 1, 9, and 11 were relatively easy. Of all students, 91%, 84%, and 78%, respectively, provided a correct response. Item 4 was the most challenging in the test: only 32% of the students responded correctly.

Regarding the justification performance, the final model was statistically significant, explaining 37% of the variance, F(14, 233) = 11.28, p < .001. In this model, prior knowledge (β = .13, p = .018) and five reading comprehension items (Item 3: β = .17, p = .005; Item 4: β = .18, p = .002; Item 5: β = .16, p = .003; Item 7: β = .24, p < .001; Item 8: β = .12, p = .026) were statistically significant predictors. Of these items, only Item 4 predicted both identification and justification performance. Two other items predicting justification performance required students to infer a word’s meaning using morphological knowledge (7) or the text context (8). One item required students to make a bridging inference across two consecutive sentences (3). Finally, one item (5) asked students to indicate which of four statements was not true. Students needed to locate the corresponding ideas across the text (all in different paragraphs) and compare the statements to the text content. The statements and corresponding ideas in the text used different expressions and successful completion of the item required inferencing. The items predicting justification performance were substantially more difficult for students than were items predicting identification performance (Item 3: 56%, Item 5: 42%, Item 7: 42%, and Item 8: 54%).

Discussion

This study examined adolescents’ credibility evaluation skills. In particular, we focused on their justification performance, specifically on how well sixth grade students can justify the author’s expertise, the author’s benevolence, and the quality of evidence. We also sought to understand how prior knowledge and basic reading skills (word recognition and reading comprehension) may contribute to students’ credibility evaluation, that is, to their identification of components (author, claim, and evidence) subject to credibility evaluation and to justification of the credibility. Consequently, this study shed further light on the relationship between credibility evaluation and lower- and higher-level reading processes. As such, it may contribute to the building of a theoretical understanding of credibility evaluation and the design of instruction that can promote students’ critical reading development.

We found that identifying the argument (i.e., claim and evidence) and the author was easy for most students, maybe because the argument structure was relatively simple (cf. Christodoulou & Diakidoy, 2020) and the author information was explicitly available. However, some students struggled to identify the correct author in online texts (see also, Coiro et al., 2015). These students confused the author with the publication venue and, for example, a person interviewed in the text.

Further, justifying credibility seemed to be more challenging for students than identifying the argument and the author. Depending on the target of the justification and the text, the proportion of the students who chose the correct justification option varied from 26% to 63%. Students performed best in justifying the author’s benevolence, with the proportion of students who chose the correct justification option varying from 58% to 81%. It should be noted that the multiple-choice options likely scaffolded students’ responses. For example, in this study, 63% of the sixth graders identified the author’s commercial intentions. In contrast, in a study that measured sixth graders’ justification performance with open-ended responses, only 19% of the sixth graders recognized or were able to express the author’s commercial intentions (Kiili, Leu, Marttunen, Hautala, & Leppänen, 2018). These results suggest that expressive tasks are more challenging than multiple-choice tasks (Sparks et al., 2021), which should be considered when interpreting results across the studies.

In line with previous results (e.g., Barzilai & Zohar, 2012; McGrew, 2021; Potocki et al., 2020), quite a few students did not attend to source features, or they did not consider whether the author’s expertise matched the topic of the text. Further, our results showed that a causal claim based on the author’s observations was particularly difficult to question, as only 25% of the students could do that. This is not surprising, given that understanding causality may be challenging even for undergraduates (List, 2023). Even though our results suggest that some credibility aspects are more difficult to justify than others, it should also be noted that on some items, the alternative justification options may have been more difficult than on others.

We found that all examined independent variables (prior knowledge, word recognition, and reading comprehension) correlated positively with identification and justification performance. However, word recognition and reading comprehension were statistically significant predictors of identification performance, whereas prior knowledge and reading comprehension were statistically significant predictors of justification performance. Thus, successful justification performance seemed to require more demanding reading processes than did identification performance. Framing these findings with Kintsch’s (1998) construction-integration model, establishing a textbase was probably sufficient for a good performance regarding the identification of the author and the argument, whereas justifying the credibility likely required the building of a situational model. Presumably, students needed to make more inferences beyond the text to justify the credibility, with prior topic knowledge supporting inferencing in this process. Noteworthy, our study was limited in considering only prior topic knowledge. It remains for future research to examine whether the role of prior knowledge would be even more substantial if other types of knowledge, such as source knowledge, were included in the examination.

Our findings are in accordance with previous studies (Forzani, 2018; Kanniainen et al., 2019) suggesting that basic reading skills, especially reading comprehension, are foundational in credibility evaluation of online texts. However, the contribution of reading comprehension to credibility evaluation may vary depending on how the former is measured (cf. Andreassen & Bråten, 2010; Keenan, Betjemann, & Olson, 2008). In addition, reading comprehension tests may include items that measure different types of reading processes, which also seemed to be the case in our study. We found that more difficult reading comprehension test items, such as inferencing beyond the text, predicted justification performance. In contrast, reading comprehension items that were relatively easy for students (with one exception) predicted identification performance. The results of this initial examination suggest that it would be valuable to examine further how different types of reading comprehension processes may contribute to credibility evaluation.

Limitations

We acknowledge that the present study has several limitations. First, the low reliability of the identification performance may be considered a limitation. This low reliability may be caused by the fact that identification performance included different types of identification (i.e., identification of the author, claim, and evidence) across four texts that represented different genres.

Second, using multiple-choice items in capturing students’ justification performance also comes with some limitations. Students were forced to choose one of the four options to justify their credibility evaluation. The provided options might not have represented their actual thinking. To mitigate this limitation, we used students’ authentic written justifications to inform the formulation of the justification options. This procedure ensured that the options would reflect, at least to some extent, students’ ways of thinking. Further, multiple-choice items may overestimate students’ justification ability because the options prompt students’ thinking. However, an advantage of multiple-choice questions in measuring credibility evaluation is that performance is not dependent on students’ writing skills (see McCarthy et al., 2022; Sparks et al., 2021).

Third, the reading comprehension test used in this study seemingly measured various types of comprehension processes, with only one or a few items measuring a specific comprehension process. Therefore, we could not make any firm conclusions about the specific types of comprehension processes that could facilitate credibility justification. Future research could profitably use reading comprehension measures focusing on specific comprehension processes, such as global coherence inferences (Jensen & Elbro, 2022).

Instructional Implications

Teachers can support the development of their students’ credibility justification skills in various ways. As reading comprehension skills seem to build an important foundation for justifying credibility, at least in online text contexts, teachers need to ensure that students have adequate opportunities to continuously practice reading comprehension, especially with tasks that require inferencing at the paragraph and text levels or even across texts. As it has been shown that reading longer texts either offline or online support comprehension skills (Torppa et al., 2020) and online reading comprehension (Kanniainen et al., 2022), fostering students reading engagement at school and home also seems important when educating critical online readers.

Because students’ prior knowledge also may facilitate credibility justification, teachers could choose text topics students already know about when practicing reasoning about credibility. Prior topic knowledge could be activated before, during, and after reading, for example by means of open-ended prompts, visualizations, or analogical reasoning (Hattan & Alexander, 2024). Especially, this may assist students who do not activate and utilize their prior knowledge spontaneously. Activation could go beyond topic knowledge, however, with teachers also activating students’ knowledge about sources.

As shown in this study, some students may need support identifying the author, claim, and evidence so that they can successfully proceed to thinking about the credibility of the author and the content of the text. In classrooms, teachers could discuss with their students the roles and responsibilities of the author, the publisher, and the interviewee, that is, how they can be considered to contribute to the online text. When reading in an authentic online environment, students may also need help locating author information. Argument knowledge, such as the form and the function of arguments, has also been shown to facilitate the evaluation of arguments (Christodoulou & Diakidoy, 2020).

Finally, students would benefit from explicit instruction about how author information can be used to justify the author’s expertise and benevolence, and why one type of evidence is stronger than another. Teachers could, for example, model thinking (see Coiro, 2011) about whether the authors’ expertise match the topic of the texts, and what ethical and/or journalistic principles (ideally) direct researchers and journalists in their work. It could be useful also to demonstrate why causal claims cannot be proved by one observation or by personal experiences. For example, students could brainstorm potential, competing explanations for what might have caused the phenomenon in question. Having students discuss and reason about credibility in small-groups and share their reasoning with the class could provide teachers with opportunities to nudge shallow reasoning toward a deeper one. As the accuracy of information cannot always be judged by examining one single text, teachers should also encourage students to confirm the accuracy of information by checking other resources (Wineburg, Breakstone, McGrew, Smith, & Ortega, 2022).

Ethical Approval

The ethical statement has been given by the Ethics Committee of the Tampere Region.

Author Contributions

CRediT: Carita Kiili—Conceptualization, Methodology, Formal analysis, Resources, Writing - original draft, Project administration, Funding acquisition; Helge I. Strømsø—Conceptualization, Methodology, Resources, Writing -review and editing; Ivar Bråten—Conceptualization, Methodology, Resources, Writing -review and editing; Jenni Ruotsalainen—Writing -review and editing; Eija Räikkönen—Methodology, Writing -review and editing.

Acknowledgement

The authors would like to thank Jari Hämäläinen, Symcode Oy for the software development of CORRE and Riikka Anttonen for helping in the data collection.

Disclosure Statement

No potential conflict of interest was reported by the author(s).

Data Availability Statement

Data available on request from the authors

Additional information

Funding

The study was funded by the Research Council of Finland (No. 324524). The work of the third author was supported by Strategic Research Council (No. 335625, No. 335727).