Student evaluations of the credibility and argumentation of online sources

Abstract

This study investigated upper secondary school students’ skills in evaluating the credibility and argumentative content of a blog text and a YouTube video. Both sources concerned child vaccination, the blog text opposing and the YouTube video supporting it. Students rated each source as credible, fairly credible or non-credible, justified their ratings, and analyzed the argumentation of both sources. Their justifications were analyzed for trustworthiness and expertise and their argument analyses for identification of the main position of the source and the reasons supporting it. Students’ justification skills proved fairly weak, and they also struggled with recognizing unbalanced argumentation. Students’ skill in analyzing the argumentation used in the sources also proved inadequate, especially in the blog text task. Overall academic achievement significantly predicted students’ credibility evaluation and argument analysis skills. The results suggest that greater emphasis should be placed on tasks involving the interpretation and analysis of online information.

KEYWORDS:

• Online sources
• multimodality
• upper secondary school
• argumentation
• critical literacy
• credibility evaluation

Introduction

The internet offers a favorable environment for disseminating inaccurate and biased information, including misinformation and disinformation (Walsh, 2010). This is possible as anyone can publish freely on the internet without subjection to any quality control mechanisms (Zubiaga et al., 2018) or traditional gatekeeper models (Metzger et al., 2010) that have been instituted to uphold the credibility standards of online information. When defining bias Walton (1991) focuses particularly on the argumentative content of the source. According to him, the basic idea of bias is a lack of appropriate balance or neutrality in argumentation, that is, a tendency to consistently and unduly favor one side over the other, ignoring the arguments on both sides. Further, Walton stresses that bias also involves an absence of critical doubt, which appears in a failure to push the viewpoint one supports, thereby implying a departure from reasoned argumentation and critical discussion. Another reason for bias may be that an arguer has some ulterior intentions or something to gain based, e.g., on some ideological or commercial interest (Kammerer et al., 2015).

With the rapid growth of electronic commerce, for example, where purchasing decisions are increasingly made in online environments, lack of objectivity in the information given about merchandise can be misleading (Xiao & Benbasat, 2015), exposing shoppers to commercial manipulation. Inaccurate or biased information is also harmful to citizens and society as it threatens democratic political processes and the democratic values that shape public policies in such life domains as health, science, and finance (Broniatowski et al., 2018; Kahne & Bowyer, 2017). For example, internet searches are commonly made for health-related information (Chen et al., 2018) despite the fact that many health and nutrition websites are inaccurate or commercially biased (see Kammerer et al., 2016) and may even contain distorted information (Chen et al., 2018). Thus, important decisions pertaining to health issues may be more strongly affected by biased and flawed argumentation advanced by manufacturers than by rational argumentation and objective comparison of alternatives.

Because biased online information may have harmful effects, protecting people from deceitful information is important. As digitalization increasingly penetrates all areas of life, the role of schools in promoting the capacity of young people to use and identify accurate and useful information becomes increasingly important (Kahne & Bowyer, 2017). Promoting students’ skills in evaluating the credibility of online information and analyzing the quality of its argumentation is of particular importance in helping students discriminate between reliable and unreliable online sources. Namely, research has shown analytical argumentation and thinking to be important for the critical evaluation of online information. Pennycook and Rand (2019) found that the ability to discern partisan fake news from real news was associated with the propensity to engage in analytical reasoning. Further, Bronstein et al. (2019) found that people over 18 with reduced analytic thinking were prone to believe in fake news.

The present study investigated upper secondary school students’ ability to evaluate the credibility and argumentative content of two online social media resources familiar to young people, i.e., a blog text, and a YouTube video. The two sources represented opposite viewpoints on a current health-related topic, child vaccination. To find out to what extent students paid attention to source bias, both sources were selected to include flawed and unbalanced argumentation favoring only one side of the question of whether it is wise to vaccinate children or not.

Literature review

Evaluation of credibility and argumentation

Credibility can be defined as the believability of a source or message and has two primary dimensions: trustworthiness and expertise (Flanagin & Metzger, 2008). Tseng and Fogg (1999) define expertise as knowledgeable, experienced, and competent and trustworthiness as well-intentioned, truthful, and unbiased.

When evaluating the credibility of online sources for expertise, various source information and source features (Bråten et al., 2018) are often of interest, including document type, when and why the document was published, whether the author’s or organization’s contact information is given, and the author’s affiliation and qualifications (Metzger, 2007; Paul et al., 2017). When evaluating trustworthiness, various content features of the web site are important, e.g., how objective, up to date and in-depth is the information provided (Metzger, 2007) and whether there are indications of source bias. Source bias may appear in unbalanced argumentation when only one side of the issue is covered and favored over the other, or in some hidden intentions or motives of the writer related to e.g. political or commercial interests (Kiili et al., 2018; Walton, 1991).

Evaluating the argumentative quality of a text requires the reader to pay attention to its argumentative structure. Toulmin (1958) defines the following main argumentative elements for analyzing the structure of argumentative texts: a claim is the argument’s conclusion, data are the facts cited as premises for the claim, and a warrant is a general operating principle bridging the data and the claim (Fulkerson, 1996, p. 18; Nussbaum, 2011). In an argumentative text, a claim or a conclusion presents the writer’s main position or message. In a high-quality argumentative text, the main claim is clearly stated and supported by relevant and sufficient premises (Fulkerson, 1996).

Argumentation plays an important role when evaluating the trustworthiness of online sources, as one should be able to assess whether the premises underlying the views presented are relevant and sufficient (Fulkerson, 1996), whether the text harbors hidden motives (e.g., economic or ideological), and whether the premises for and against the claim receive equal weight in the source.

The terms grounds (Osborne et al., 2004), evidence (Forzani, 2016) and reason (Means & Voss, 1996) are also used when referring to elements of an argument justifying a claim or a conclusion. In this study, the term “reason” is used to refer to elements of arguments supporting the claim or conclusion, and “argumentation perspective” to refer to categories of reasons representing a specific perspective, such as a financial, ethical or health-related perspective. Xu and Yao (2015) state that the justifications or reasons representing various argumentation perspectives indicate the breadth of argumentation, a further necessary criterion for high-quality argumentation. Breadth of argumentation, or space of debate as van Amelsvoort et al. (2007) put it, is defined as the number of topics and subtopics mentioned that relate to the issue discussed: The higher the number of topics and subtopics related to the arguments used in the discussion, the higher the argumentative quality of the discussion.

Previous research on evaluation of online information

It is easily assumed that because young people are fluent in using the internet and social media, they are equally savvy about what they find there. However, recent research results show the opposite. Coiro et al. (2015) found that 7^th graders were unable to justify whether the author of an online source was an expert in the field the source represented. Further, the Stanford History Education Group (2016) showed that students at different educational levels had poor skills in evaluating information delivered through social media channels, and thus were at risk for being duped by disinformation, including flawed argumentation. For example, Miller and Bartlett (2012) in their survey (n = 509) of primary and secondary school teachers reported that students commonly struggle to recognize biased information and may use information that has been deliberately produced to be misleading. Similarly, Kiili et al. (2018) found that only about half of their sample of 6^th grade elementary school students questioned the credibility of the commercial online source and only about a fifth fully recognized commercial bias. Similarly, Fogg et al. (2003) asked somewhat older regular web users (n = 2 684; average age 39.9) to comment on the credibility of preselected websites representing various content categories. Only 11.6% of the participants’ credibility comments referred to information bias.

Kiili, Laurinen, and Marttunen (2008) found that upper secondary school students seldom evaluated the credibility of online information and that when they did, they mainly attended to such aspects as the publisher, author or expert interviewed in the source. Cognitively demanding evaluation strategies, such as the evaluation of argumentation or comparison of texts, were rarely used. At college level, McClure and Clink (2009) examined English composition students’ research essays to determine the amount of attention students gave to analyzing and crediting online sources, particularly their timeliness, authority, and bias. They found that students not only struggled with understanding bias but also ignored it when it hampered their writing.

Previous research has shown that students' critical thinking has a positive association with their academic achievement (e.g., Fong et al., 2017; Ghanizadeh, 2017; Marin & Halpern, 2011). Similarly, university applicants' argumentative writing skills predicted their academic achievement in university (Preiss et al., 2013). These findings stress the importance of academic achievement in terms of students’ argumentation and critical thinking. In this study we explored this issue further by studying whether upper secondary school students' academic achievement is associated with their credibility evaluation and argument analysis skills.

Prior studies also suggest that girls perform better than boys in critical evaluation of online sources (Forzani, 2016; Taylor & Dalal, 2017), prior knowledge on a topic enhances students' critical evaluation of sources on similar topics (Forzani, 2016), and that offline and online reading skills are intertwined but their relationship remains unclear (Leu et al., 2013; Kiili et al., 2018). Further, prior knowledge occupies an important role in readers’ activities during offline reading comprehension (e.g., Means & Voss, 1985) but in online reading contexts its role is more complex (Coiro, 2011). To explore these issues further, this study investigated to what extent students’ performance in the critical and argumentative evaluation of online sources can be explained by their gender, prior knowledge on the topic, and offline reading fluency.

Aim and research questions

Many studies have shown that students commonly struggle in evaluating the credibility of online sources and recognizing bias. Deficiency in credibility evaluation is a serious problem as, e.g., bloggers and video bloggers have a lot of influence particularly on young people who often adopt their ideas despite the absence of guarantees on the trustworthiness, objectivity, and unbiased nature of information disseminated in social media. Thus, further knowledge is needed on how students evaluate online sources low in credibility.

Students’ recognition of bias in online information has been widely investigated. However, to our knowledge, no studies exist in which students have evaluated the credibility of argumentatively unbalanced online sources representing different modalities (text, video) and addressing the same topic from opposite viewpoints (for and against). Such a comparative study may further understanding of whether young people differ in how they deal with unreliable information presented through different social media and whether they are more likely to be misled by erroneous and unobjective information presented via one type of social media rather than another. Further, this information is needed in developing pedagogy to help future citizens recognize inaccurate information and fake news (Vosoughi et al., 2018).

This study investigated how students deal with two online social media sources frequently used by young people: a written blog and a multimodal video. Upper secondary school students were given a blog text and a YouTube video, one supporting and the other opposing the same currently contested topic (child vaccination). These online sources were deliberately selected for their unbalanced argumentation on the topic and hence questionable credibility. Both sources lacked credibility also in that the author of both sources was not an expert in the field (vaccination) and that they did not contain any references to support the presented arguments. Students’ skills in evaluating these online sources for argumentative content and credibility were explored. Also examined was whether students’ skills can be explained by gender, prior knowledge on the topic, reading fluency, and previous academic achievement. The research questions were:

1. How skilled are upper secondary school students in evaluating the credibility of unreliable online sources?

   • 1.1 How do students rate the credibility of unreliable sources?

   • 1.2 How do students justify the lack of credibility of unreliable sources?

   • 1.3 Do students identify unbalanced argumentation?

2. How skilled are upper secondary school students in analyzing the argumentative content of unreliable online sources?

3. To what extent are students’ skills explained by gender, prior knowledge on the topic, reading fluency, and previous academic achievement?

Method

Subjects

Participants were 404 upper secondary school students (94.6% aged 17) from seven schools in Finland. There were 60.4% female students and 38.9% male students (missing information 0.7%). Most (79.2%) of the participants were students in three large urban upper secondary schools and the rest (20.8%) students in four small and middle-sized schools in semi-urban and rural areas.

A written consent to participate in the study was obtained from the parents of underage students. Our research was approved by all the participating upper secondary schools and included only minimum risk for participants. According to the Finnish standard for institutional review board approvals, a statement by the Ethical Board of the University was not required.

Study context

This study is a part of a larger research project on Finnish upper secondary school students’ argumentative online inquiry competencies. In the project, students completed an online task in which they composed an argumentative letter to imaginary Finnish members of parliament on the advantages and disadvantages of child vaccination aimed at informing their legislative work on this issue. The online task followed an online research and comprehension framework (Leu et al., 2013), including sub-tasks on the information search, evaluating, synthesis and communication. This article reports the results of the analyses of a sub-task on students’ evaluations of the credibility and argumentative content of two online sources utilizing the evaluation component of the framework by Leu et al. (2013).

The two sources, a blog text and a multimodal YouTube video, viewed child vaccination from opposite standpoints. The Finnish blog text (3 pages, 967 words) presented several potential disadvantages of vaccination by appealing to various side effects of vaccines, the financial benefits of vaccination for the pharmaceutical industry, and emotional arguments by describing situations in which parents have reported harmful effects of vaccines on their children. The blog text concluded that children should not be vaccinated. In contrast, the YouTube video presented arguments for vaccination, concluding that it is wise to vaccinate children. In the video, two US comedians (Penn and Teller) argued for vaccination by means of humor, music, sound effects, and visual demonstrations such as gestures and motion. The comedians also cited statistics on both the effectiveness of vaccinations and their minimal injurious effects and described the potential negative effects of not vaccinating children. The video, in English with Finnish subtitles, lasted for 1.5 minutes.

Data collection

Prior knowledge and reading fluency

In this study, students’ prior knowledge on the child vaccination was assessed with ten statements, three of which were correct and seven false. The statements were selected based on both current scientific knowledge and common misinformation disseminated by anti-vaccination movements. To strengthen content validity, two university lecturers in health sciences and a medical expert reviewed the statements. Students were asked to select the three statements they thought were correct. Students’ prior knowledge scores varied from 0 to 3 depending on the number of correct statements selected.

Reading fluency was measured by a word chain test (Holopainen et al., 2004). The task was to separate as many of a series of 100 words, written in four-word clusters without spaces in between, as possible in 90 seconds (Kiuru et al., 2011).

Credibility evaluation and argument analysis tasks

To evaluate the credibility of the online sources the students completed two tasks, one for the blog text and one for the YouTube video. In both tasks, the students first rated the credibility of the source by answering a multiple choice question (How credible do you think the text/video is?) by selecting one of three responses (credible, fairly credible, not credible), and then justifying their credibility evaluation in writing by answering the question “Justify your choice, why do you think this?.”

Students also completed two tasks on the argumentative content of the online sources, one for the blog and one for the video. In both tasks, the students wrote down the main position on child vaccination they identified in the source and the most important justifications for that position given in the source.

Data analysis

Students’ credibility evaluation skills

First, the frequencies of the students’ responses (n = 404) to the multiple-choice questions on the credibility of each online source were calculated. Next, the students’ justifications for their credibility ratings (n = 403) were analyzed for trustworthiness and expertise. On trustworthiness, students scored points when they appealed to content-related issues: use of citations, research-based information, and unbalanced argumentation (Tseng & Fogg, 1999; Metzger, 2007; Walton, 1991). On expertise, the students scored points when they cited external characteristics of the source, e.g., author attributes, form of the web source, and organization(s) affiliated with the web page (Metzger, 2007).

As both sources included unbalanced argumentation on the topic their credibility was principally deemed questionable. Consequently, students scored 0 points if they had not questioned the credibility of the source, i.e., selected the response option “credible”. When they had questioned credibility, i.e., chosen the options “fairly credible” or “not credible”, the response was scored from 1 to 4 depending on how well the lack of credibility had been justified. No justifications scored 1 point and one or more justifications, relating either to expertise or to trustworthiness, scored 2 points. Identification of unbalanced argumentation was emphasized in the scoring. Thus, 3 points was scored when lack of credibility had been justified by appealing to unbalanced argumentation. Identification of unbalanced argumentation is illustrated in the following two examples of students’ justifications (see Table 1): The text is based only on the disadvantages of vaccination and totally ignores its advantages (Student 428, Blog text); The issue was treated only from one side and the attitude for not using vaccines was initially negative (Student 174, YouTube video). Justifications including both expertise and trustworthiness, although without reference to unbalanced argumentation, also earned 3 points. The highest score (4 points) was given when the justifications included, at the least, both expertise and reference to unbalanced argumentation (Table 1).

Table 1. Examples of scoring students’ responses to the credibility evaluation tasks.

		Examples of student responses
Score	Scoring criteria	Blog text task	YouTube video task
0	Credibility of source not questioned	The facts prove that the claims of the author are credible. (St. 390)	The video was convincing. (St. 263)
1	Credibility questioned without relevant justifications for the lack of credibility	Lots of facts. (St. 417) Everybody can find such things on the internet because it is so wide (St. 151)	The video is quite credible because it includes facts and exact figures. (St. 37)
2	Credibility questioned, and justification(s) relating either to expertise (E) or trustworthiness (T) presented	Because this is a blog (E) and the writer is necessarily not an expert (E) in this field (St. 219) No references (T) listed at the end of the text, and the studies were somewhat vague (St. 290)	Penn and Teller are magicians, not vaccination guys (E) (St. 422) There are no references (T) presented on the video. (St. 199)
3	Credibility questioned, and justifications relating both to expertise (E) and trustworthiness (T) presented, or unbalanced argumentation (UA) identified	The text is based only on the disadvantages of vaccination and totally ignores its advantages (UA) (St. 428) Because the text is a blog written by one person (E), an opinion without reliable references (T) (St. 409)	The issue was treated only from one side (UA) and the attitude for not using vaccines was initially negative (St. 174)
4	Credibility questioned, and unbalanced argumentation (UA) identified, and justification(s) relating to expertise (E) presented	The source is a website of a natural products company, and the text is a blog (E) which makes is primarily rather subjective. Discussion on the issue is very one-sided (UA), and different approaches to the issue are not presented. (St. 436)	I think like this because the video does not tell whether the persons are professionals or not (E), or whether they are only expressing their own opinion. Although the video also includes facts, no references (T) are used. The video is solely about the advantages of vaccines, nothing about disadvantages (UA), although they must certainly also exist. (St. 171)

Note. St. = Student; T = Justification relating to trustworthiness; E = Justification relating to expertise; UA = Justification relating to unbalanced argumentation; The core words guiding the scoring are italicized.

To test the reliability of the analysis criteria of students’ justifications, 15% of the data were selected at random and analyzed by two independent raters, reaching 82% agreement (Cohen’s kappa, ĸ =.74).

Students’ argument analysis skills

Analysis of students’ skills to analyze the argumentation of the blog text and the YouTube video proceeded in two phases. First, the researchers analyzed the argumentation of these two online sources. Next, the students’ responses to the argument analysis tasks were compared with the argumentation in the sources.

Argumentation of the online sources

First, all the reasons against vaccination in the blog text (42 reasons) and all those for vaccination in the video (5 reasons) were identified. The identified reasons were then classified into argumentation perspectives on a data-driven basis (Hsieh & Shannon, 2005).

For the blog text, the argumentation perspectives (an example of a reason in parenthesis) were: 1) Consequences of vaccination (Vaccination results in serious, even fatal injury); 2) Effectiveness of vaccination (Going through things notices that vaccinations do not protect from diseases); 3) Consistency of vaccination (Viral vaccines can never be safe); and 4) Support in problematic situations (Children falling ill from vaccinations are given help by official bodies). Similarly, for the video, the argumentation perspectives were: 1) Consequences of vaccination (Unvaccinated people are more likely to contract diseases); 2) Effectiveness of vaccination (Before Salk’s vaccine there were approximately 58,000 cases of polio).

Analyzing students’ responses

Students’ skills in analyzing the argumentation in the source were scored based on their responses to each task (397 for the blog and 403 for the video). Students who failed to correctly identify the position taken in the source scored 0 points. Students who correctly identified the source position scored from 1 to 4 depending on the quality of their identification (position quality) and how many argumentation perspectives they found to support that position.

Position quality was analyzed by investigating how well the position identified by a student corresponded to the source position, i.e., against vaccination in the blog and for vaccination in the video. Position quality indicated how well a student had understood (partially vs. fully) the main message of the source. In responses indicating partial understanding the source position was correctly identified but either only implicitly embedded in the response or expressed in the form of a descriptive claim by referring, for example, to specific harms or benefits of vaccines. Examples of descriptive claims were “Vaccines cause autism” (Student 416, Blog text) and “Vaccines protect people from severe diseases” (Student 95, YouTube video). Full understanding of the main message of the source was indicated by responses in which the source position was expressed in a form of a correct conclusion expressed in the source. Examples of correct conclusions were “Children should not be vaccinated” (Student 335, Blog text) and “It is wise to vaccinate children” (Student 217, YouTube video).

The minimum requirement for a score of 1 in the blog task was that the student’s response included a position on child vaccination corresponding to that presented in the source (i.e., against vaccination, Table 2). Students scored 2 when the source position (SP) was correctly identified and justified from two argumentation perspectives (AP), as illustrated in the following response by student 296 (analysis markings of the authors in parentheses): “The author is clearly against vaccinations (SP) and presents many justifications for her stand. She states that more children vaccinated against whooping cough contracted the disease than non-vaccinated children (AP 1, Effectiveness of vaccination). She also extensively describes the harm caused by vaccinations” (AP 2, Consequences of vaccination). Students scored 2 also when the response included a correct conclusion justified from one argumentation perspective. Score 3 was given when the source position was correct and justified from three argumentation perspectives, or a correct conclusion was justified from two argumentation perspectives. Score 4 was given when the response included a correct conclusion justified from at least three argumentation perspectives (Table 2).

Table 2. Examples of scoring of students’ responses to the blog text argumentation analysis task.

Score	Student response	Interpretation
0	The author thinks vaccination is useful but not necessary. She justifies her opinion by facts. (St. 122)	Source position (Vaccination is useful) not correctly identified
1	The author finds vaccines dangerous and that one should take them with a pinch of salt. (St. 303)	Source position (against vaccination) correctly identified (Vaccines are dangerous).
2	The author is clearly against vaccinations and presents many justifications for her stand. She states that more children vaccinated against whooping cough contracted the disease than non-vaccinated children. She also extensively the describes the harm caused by vaccinations (St. 296)	Source position correctly identified (against vaccination) and justified from two argumentation perspectives: Effectiveness of vaccination (Vaccinated children contracted the disease they were vaccinated against*) Consequences of vaccination (Vaccinations cause harm*)
3	The author thinks vaccines are completely useless and might even be dangerous. She justifies her claim by examples on how vaccines have several injurious effects and that they do not always protect people from diseases they are vaccinated against. (St. 337)	Correct conclusion (Vaccines should not be used) justified with two argumentation perspectives: Consequences of vaccination (Vaccines have many different harmful effects*) Effectiveness of vaccination (Vaccines do not always protect people from diseases they are vaccinated against*)
4	The author thinks vaccines are harmful and should not be used. She justifies her claim by giving examples of people who have contracted the disease they were vaccinated against. She explains why she thinks that viral vaccines are dangerous by describing how they are produced. At the end she also tells about a child who became autistic after vaccination, and about a child who had narcolepsy after vaccination. (St. 385)	Correct conclusion (Vaccines should not be used) justified with three argumentation perspectives: Effectiveness of vaccination (Vaccines are not effective, because vaccinated people contracted the disease*) Consistency of vaccination (Viral vaccinations are dangerous because of the production process*) Consequences of vaccination (A child became autistic after vaccination and another child had narcolepsy*)

Note. St. = Student; * = Authentic reason included in the blog text.

Students scored 1 on the YouTube video task when they correctly identified the source position taken on child vaccination (i.e., for vaccination, see Table 3). Students scored 2 when either the source position was correctly identified and justified from one argumentation perspective, or the conclusion was correct but lacked justifications. Score 3 was given when the source position was correct and justified from two argumentation perspectives, or the conclusion (CO) was correct and justified from one argumentation perspective (AP). The latter option is illustrated in the response by student 354 (analysis markings of the authors in parentheses): “The main message is that vaccination is not useless (CO). The most important justification is the survival of vaccinated children” (AP, Effectiveness of vaccination). Students scored 4 when their response included a correct conclusion justified from both argumentation perspectives (Table 3).

Table 3. Examples of scoring of students’ responses to the YouTube video argumentation analysis task.

Score	Student response	Interpretation
0	The key point is that vaccines are complete bullshit :) (St. 134)	Source position not correctly identified (Vaccines are bullshit)
1	Vaccines protect against many diseases and deaths (St. 115)	Source position (for vaccination) correctly identified (Vaccines protect against diseases).
2	Vaccines do not cause autism and vaccines save people from bad diseases. The most important justifications were the use of visuals showing how vaccinated people can protect themselves from diseases and demonstrating the number of deaths from these diseases when not vaccinated. (St. 95)	A descriptive claim (Vaccines save people from bad diseases) justified from one argumentation perspective: Effectiveness of vaccination (Vaccines are effective, because they protect people from diseases*)
3	The main message is that vaccination is not useless. The most important justification is the survival of vaccinated children. (St. 354)	Correct conclusion (Vaccination is not useless) justified with one argumentation perspective: Effectiveness of vaccination (The most important justification is the survival of vaccinated children*)
4	The main point is that it is stupid not to vaccinate a child. Therefore, the video supports vaccines. The opinion was justified by listing many diseases against which vaccinations are used and saying that one can get the disease if not vaccinated. However, the video mentions that research exists claiming that vaccinations cause autism. However, this claim was completely refuted in the video. (St. 17)	Correct conclusion (It is stupid to not vaccinate a child) justified with two argumentation perspectives: Effectiveness of vaccination (Vaccinations protect children from many diseases*) Consequences of vaccination (Vaccinations are claimed to cause autism, which is refuted in the video*)

Note. St. = Student; * = Authentic reason included in the video.

To test the reliability of the analysis criteria for both the blog and the video, two independent raters classified 15% of the data selected at random, reaching 87% agreement (Cohen’s kappa, ĸ =.79).

Variables used in the study

The dependent variables used in this study were formed based on the students’ credibility ratings, credibility evaluation, identification of unbalanced argumentation, and argument analysis for each source (Table 4).

Table 4. Variables used in the study.

Variable	N	Scoring range
Dependent skill variables
Credibility rating for the blog	403	1–3
Credibility rating for the video	403	1–3
Credibility evaluation for the blog	403	0–4
Credibility evaluation for the video	403	0–4
Identifying unbalanced argumentation in the blog	403	0 = No; 1 = Yes
Identifying unbalanced argumentation in the video	403	0 = No; 1 = Yes
Argument analysis for the blog	397	0–4
Argument analysis for the video	403	0–4
Independent variables
Grades
Philosophy	273	4–10
Psychology	337	4–10
Health education	311	4–10
Mother tongue and literature	344	4–10
Mathematics	331	4–10
History	342	4–10
Other
Reading fluency	404	0–100
Prior knowledge	404	0–3
Gender	401*

Note. * Females, n = 244 (60.6%); Males, n = 157 (39.2%).

Students’ most recent grades in various upper secondary school subjects, performance in the word chain test, prior knowledge on vaccination, and gender were used as independent variables (Table 4). In Finland, academic performance in various subjects in upper secondary schools is rated from 4 to 10. Moreover, in the Finnish course-based school system at upper secondary level (European Commission, 2019), both schools and students have considerable freedom in organizing studies. Hence, the number of courses taken in different subjects may vary considerably between students, as was the case here. Moreover, some students did not consent to the use of their school grades for research purposes, which increased the amount of missing data. The proportion of missing data for the grades ranged from 15% to 32%.

Statistical analyses

To answer research questions (RQ) one and two, descriptive statistics on the level of students’ credibility evaluation and argument analysis skills were used. Additionally, Wilcoxon’s signed ranks test was applied to compare students’ credibility ratings for the blog text and video, and McNemar’s test was used to compare how students identified unbalanced argumentation in the sources (Davis, 2013). Furthermore, paired samples t-test was applied to compare differences between the blog text and the video in students’ overall performance in the credibility assessments and argument analysis skills (Davis, 2013).

To answer RQ 3, Structural Equation Modeling (SEM) was applied (Kline, 2011). All SEM analyses were conducted using Mplus 7.4. Maximum likelihood estimation with robust standard errors (MLR) was used as an estimation method, as the variables showed slight non-normality (skewness values from −.68 to .45 and kurtosis values from −.80 to 1.20).

In the analysis, Overall academic achievement, Prior knowledge of the topic, Reading fluency and Gender were independent variables, and Argument analysis and Credibility evaluation skills dependent variables. In addition, school grades were indicators for the bi-factor model: Overall academic achievement was modeled as a generic factor for which all the individual grades were indicators. To complement the generic factor, specific factors for each individual grade were estimated to model variance not accounted for by the general factor.

The following fit indices and cutoff values for acceptable fit of the model were used: χ²-test, ns, p > .05, Root Mean Square Error of Approximation (RMSEA) value under .06, and Tucker-Lewis Index (TLI) and Comparative Fit Index (CFI) values over .95 (Hu & Bentler, 1999). Effect sizes used for the repeated measures tests were reported as Cohen’s d (Cooligan, 2017). For the non-parametric tests, the effect sizes were transformed to Cohen’s d. For the SEM-analyses, squared r’s were calculated.

Results

Students’ credibility ratings of unreliable online sources

More than a half of the students (59.1%) considered the blog text fairly credible, slightly above one-third (37.5%) not credible, and only a few (3.5%) credible. Similarly, the majority (53.1%) judged the video to be fairly credible and 38.2% non-credible. In contrast to the blog, nearly a tenth (8.7%) of the students considered the video credible. The difference between the students’ credibility ratings for the blog and the video was not statistically significant (Z = −1.20, p = .232).

Students’ skills in evaluating the credibility and analyzing the argumentative content of unreliable online sources

Students’ skill in evaluating the credibility of the online sources proved rather weak. Although the students commonly questioned the credibility of both sources, they justified the non-credible nature of the sources poorly. More than a half of the students, 50.3% for the blog and 58.8% for the video, offered no justifications or appealed only to either authority or trustworthiness (1-2 points, Table 5). Students’ performance on the blog (M = 2.4) and video (M = 2.1) tasks differed significantly (t = 4.68, df = 402, p < .001, d = .29).

Table 5. Level of credibility evaluation and argument analysis for the blog text and the YouTube video.

	0 points		1 point		2 points		3 points		4 points		M	SD
Skill variable	f	%	f	%	f	%	f	%	f	%
Credibility evaluation for the blog	14	3.5	55	13.6	148	36.7	127	31.5	59	14.6	2.4	1.0
Credibility evaluation for the video	35	8.7	32	7.9	205	50.9	108	26.8	23	5.7	2.1	1.0
Argument analysis for the blog	5	1.3	130	32.7	168	42.3	83	20.9	11	2.8	1.9	0.8
Argument analysis for the video	4	1.0	83	21.0	124	31.3	145	36.6	40	10.1	2.3	1.0

As in the credibility evaluation, students’ skill in analyzing the argumentation in the online sources was also fairly weak. This appeared especially in the blog text task. The majority (76.3%; points 0-2 in total) failed to identify the correct position taken in the text (0 points), or identified it but with either no or one justification (1 point), or found at most two of the four argumentation perspectives (2 points; Table 5). Further, only 2.8% of students correctly identified the main conclusion and several justifications for it (4 points; Table 5). Students performed somewhat better in the video task, as the corresponding proportions were 53.3% (0 − 2 points) and 10.1% (4 points). The difference between the blog (M = 1.9) and video (M = 2.3) was statistically significant (t = −7.14, df = 395, p < .001, d = .47).

Students’ identification of unbalanced argumentation

The study also investigated whether students had identified unbalanced argumentation as a justification for the non-credible nature of the sources. The results showed that this was mentioned by only slightly more than a third of the students (37.4%) in the blog text and about a fifth (21.0%) in the video. The difference between the blog and the video was significant (χ²(1) = 31.06, p < .001, d = .57).

Factors explaining students’ skills in credibility evaluation and argument analysis

The structural equation model describing the factors explaining students’ argument analysis and credibility evaluation skills, and its parameter estimates, is presented in Figure 1. The fit indices of the model were: χ²(51) = 66.31, p = .073, RMSEA = .03, CFI = .98, TLI = .97. Furthermore, all school subjects had significant and reasonable loadings, ranging from .64 to .73, for the Overall academic achievement factor. Correlations between independent variables were estimated as suggested by the modification indices.

Figure 1. Standardized parameter estimates (significant at the.05 level) for the structural equation model explaining students’ skills in argument analysis and credibility evaluation.

Note. Ovals to the left of the grades refer to the specific factors for each grade (e.g., Phil_SF = specific factor for philosophy grade).

The structural equation model (Figure 1) shows that, for both online sources, students’ overall academic achievement was the only significant predictor of their argument analysis and credibility evaluation skills. However, its explanatory power for the video was rather weak, as it predicted only 2.9% to 4.2% of the variances for the argument analysis and credibility evaluation tasks. Instead, overall academic achievement was a somewhat stronger predictor of students’ skill in evaluating the credibility of the blog text, explaining 12.3% of the variance. Finally, the scores for the credibility evaluation of the blog text correlated positively (.24) with the scores for the credibility evaluation of the video.

Discussion

Although the students in this study commonly questioned the credibility of the online sources, their justifications for the non-credible nature of the sources were fairly weak, especially in the case of the YouTube video. As many as 67.5% of students either presented no justifications or appealed only to either authority or trustworthiness. Students also seemed to struggle in identifying unbalanced argumentation, as only slightly more than a third for the blog text and about a fifth for the video mentioned unbalanced argumentation as a reason for the non-credibility of the source. This result supports previous findings on students’ difficulty in recognizing bias in online sources at the primary and secondary levels (Kiili et al., 2018; Miller & Bartlett, 2012), and at college level (McClure & Clink, 2009).

To understand the weakness of students’ justifications for the non-credibility of the YouTube video, it is important to note that the video dealt solely with the advantages of vaccination, and thus echoed to the official stand and a positive public attitude toward vaccination (e.g., The National Institute for Health and Welfare, 2019). The students probably found it unnecessary to put forward reasons for this generally accepted stand despite the obviously one-sided nature of information source. Another explanation may be myside bias (Greene et al., 2019; Perkins, 1989), that is, a tendency to prefer one’s own position on an issue and ignore evidence for the opposite position. van Strien et al. (2016) found that students seem to prefer, and to evaluate as more credible, information consistent with their own attitudes. Furthermore, research on motivated reasoning (e.g., Ditto et al., 1998; Kunda, 1990) suggests that pro-attitudinal information will be evaluated less thoughtfully than information that runs contrary to one's beliefs and attitudes. This is especially true for social-scientific issues such as vaccination (Sinatra et al., 2014). People seem also to evaluate information differently if it comes from entertainment media compared to news media; entertainment media facilitate fairly stable attitudes, such as stereotypical or ideology-based judgments (Kim & Vishak, 2008).

Students’ skill in analyzing the argumentation contained in the online sources also proved insufficient, especially in the blog text task. This result is in line with previous findings that students have difficulties in identifying and evaluating arguments in linear texts (e.g., Larson et al., 2004; Marttunen et al., 2005). Although the students’ analysis of the argumentation of the video was also weak, it was nevertheless better than that of the blog text. One reason for this may be that the argumentative structure of the video was simpler than that of the blog text: the video contained far less reasons than the blog and only two argumentation perspectives in support of its main position in contrast to the four argumentation perspectives in the blog text. The multimodal nature of the video may also have affected the result. Whereas the argumentation in the blog was mainly expressed via linear text, that in the video comprised both verbal and nonverbal communication, such as visuals, motion, and voice, used simultaneously to express its content. The multimodality of the video may have helped the students to understand its content and analyze it (see Lee & List, 2019). Tseronis (2018) states that visuals and other nonverbal means can be seen as argumentatively relevant because they can simply depict the argument, or elements of the propositions that constitute the argument, and convey the meaning of the source in more diverse ways than are possible with plain text.

Students’ lack of skills in analyzing argumentation and the credibility of online sources may, on the other hand, mirror a ‘why bother’ attitude stemming from skeptical or even cynical views (Vraga & Tully, 2021) that much online information is somehow biased in any case. Skepticism toward, e.g., politics and media in general is, however, considered a ‘healthy’ trait as it includes doubt and a questioning of certainty that encourages participation in a democratic society (Vraga & Tully, 2021) and enhances critical discussion on societal issues. For example, Maksl et al. (2015) found that highly news media literate teens were more skeptical of the media and more motivated to consume news than their less news literate counterparts. Therefore, one objective of education should be to enhance students’ ability to engage actively in society by teaching them to be sufficiently critical when searching, evaluating, and utilizing online information.

In educational settings, students often evaluate online information through argumentative writing. Wolfe and Britt (2008) found that when students prepare themselves for essay writing, for example, by taking notes that fail to include information incongruent with their own opinions, their essays tend to be one-sided. Students seem to have misconceptions about good argumentation and argumentative writing, possibly thinking that sound argumentation involves generating arguments only in support of one’s own position. However, high quality argumentation also requires analyzing information from the other side and combining arguments and counterarguments for a final position (Nussbaum & Schraw, 2007). For this reason, developing students’ understanding of more sophisticated and versatile argumentation thinking is an important educational goal that should be accorded more attention to prevent students being misled by inaccurate and flawed information. Previous studies (Bronstein et al., 2019; Pennycook & Rand, 2019) have shown, for example, that deficiencies in analytical thinking and reasoning are related to believing in biased information such as fake news.

The results also showed that students’ previous performance in various individual school subjects did not explain their skills in either evaluating the credibility or analyzing the argumentative content of the two online sources. However, students’ overall academic achievement was found to be significantly associated with these skills. This result supports the study by Utriainen et al. (2017), who found that university applicants' overall performance in the matriculation examination predicted their argument analysis skills. The present finding that students’ overall academic achievement, but not their achievement in individual school subjects, prior knowledge on the task topic, or reading fluency, explained their evaluation and analyzing skills (cf. Kiili et al., 2018; Leu et al., 2015) refer to the generic nature of credibility evaluation and argument analysis skills. However, it must be noted that, as in the study by Utriainen et al. (2017), the proportions of explanations remained rather low.

Owing to the rapid increase in internet use both in and out of school, it is very likely that current teaching practices do not yet adequately support students’ learning of the skills they need when they work internet-sourced knowledge. This concern is supported by the weak associations found in this study between students’ previous school achievement and their credibility evaluation and argument analysis skills with online sources. These findings suggest that tasks in which students interpret and analyze online information presented in various modes, both written and multimodal, should be more integrated into the instruction given in different subjects in schools.

Limitations

This study was carried out in an authentic context as a part of students’ normal schoolwork in upper secondary school classrooms. Hence, the participating schools were not randomly selected and the data are not necessarily representative of all Finnish upper secondary school students. Thus, the results cannot be generalized beyond the present sample. Further, another limitation of the study is that arguments and information presented were not counterbalanced across the blog and the video, due to the authentic study situation in schools. Same task repeated, although using different medium, would have possibly diminished students’ motivation to complete the task. The task used in this study was also a part of a large and time-consuming online inquiry task, and thus we did not find it possible to increase the students’ workload to complete the task.

Further, it should also be noted that the two online sources used in this study differed in mode, language, length, and argumentative content which complicates their comparison. To facilitate comparison of the sources, the focus was on only the verbal argumentation included in them, and thus many multimodal arguments based on, e.g., visuals, sounds, and gestures in the video were not taken into account. However, the comparison of the sources should be treated with caution. The study, nevertheless, opened interesting topics for further research to compare the credibility and argumentation of multimodal and text-based online sources.

Conclusion

Limitations aside, this study showed that evaluating the credibility and analyzing the argumentative content of two online sources commonly used among young people in social media—a written blog text and a YouTube video—was poorly accomplished by many students. The fact that many major life decisions are made based on information acquired from the internet (Horrigan & Rainie, 2006) exposes many individuals to the risk that these decisions are likely to be based on inaccurate or even flawed information. Thus, critical evaluation of information should be increasingly emphasized in schools to promote students’ skills in differentiating reliable from inaccurate information and to protect future citizens from being misled by the disinformation increasingly encountered on the internet.

Disclosure statement

None of the authors have conflicts of interest to declare.

Additional information

Funding

Academy of Finland (Project 285 817).