Student evaluations of the credibility and argumentation of online sources

Figures & data

Table 1. Examples of scoring students’ responses to the credibility evaluation tasks.

		Examples of student responses
Score	Scoring criteria	Blog text task	YouTube video task
0	Credibility of source not questioned	The facts prove that the claims of the author are credible. (St. 390)	The video was convincing. (St. 263)
1	Credibility questioned without relevant justifications for the lack of credibility	Lots of facts. (St. 417) Everybody can find such things on the internet because it is so wide (St. 151)	The video is quite credible because it includes facts and exact figures. (St. 37)
2	Credibility questioned, and justification(s) relating either to expertise (E) or trustworthiness (T) presented	Because this is a blog (E) and the writer is necessarily not an expert (E) in this field (St. 219) No references (T) listed at the end of the text, and the studies were somewhat vague (St. 290)	Penn and Teller are magicians, not vaccination guys (E) (St. 422) There are no references (T) presented on the video. (St. 199)
3	Credibility questioned, and justifications relating both to expertise (E) and trustworthiness (T) presented, or unbalanced argumentation (UA) identified	The text is based only on the disadvantages of vaccination and totally ignores its advantages (UA) (St. 428) Because the text is a blog written by one person (E), an opinion without reliable references (T) (St. 409)	The issue was treated only from one side (UA) and the attitude for not using vaccines was initially negative (St. 174)
4	Credibility questioned, and unbalanced argumentation (UA) identified, and justification(s) relating to expertise (E) presented	The source is a website of a natural products company, and the text is a blog (E) which makes is primarily rather subjective. Discussion on the issue is very one-sided (UA), and different approaches to the issue are not presented. (St. 436)	I think like this because the video does not tell whether the persons are professionals or not (E), or whether they are only expressing their own opinion. Although the video also includes facts, no references (T) are used. The video is solely about the advantages of vaccines, nothing about disadvantages (UA), although they must certainly also exist. (St. 171)

Note. St. = Student; T = Justification relating to trustworthiness; E = Justification relating to expertise; UA = Justification relating to unbalanced argumentation; The core words guiding the scoring are italicized.

Table 2. Examples of scoring of students’ responses to the blog text argumentation analysis task.

Score	Student response	Interpretation
0	The author thinks vaccination is useful but not necessary. She justifies her opinion by facts. (St. 122)	Source position (Vaccination is useful) not correctly identified
1	The author finds vaccines dangerous and that one should take them with a pinch of salt. (St. 303)	Source position (against vaccination) correctly identified (Vaccines are dangerous).
2	The author is clearly against vaccinations and presents many justifications for her stand. She states that more children vaccinated against whooping cough contracted the disease than non-vaccinated children. She also extensively the describes the harm caused by vaccinations (St. 296)	Source position correctly identified (against vaccination) and justified from two argumentation perspectives: Effectiveness of vaccination (Vaccinated children contracted the disease they were vaccinated against*) Consequences of vaccination (Vaccinations cause harm*)
3	The author thinks vaccines are completely useless and might even be dangerous. She justifies her claim by examples on how vaccines have several injurious effects and that they do not always protect people from diseases they are vaccinated against. (St. 337)	Correct conclusion (Vaccines should not be used) justified with two argumentation perspectives: Consequences of vaccination (Vaccines have many different harmful effects*) Effectiveness of vaccination (Vaccines do not always protect people from diseases they are vaccinated against*)
4	The author thinks vaccines are harmful and should not be used. She justifies her claim by giving examples of people who have contracted the disease they were vaccinated against. She explains why she thinks that viral vaccines are dangerous by describing how they are produced. At the end she also tells about a child who became autistic after vaccination, and about a child who had narcolepsy after vaccination. (St. 385)	Correct conclusion (Vaccines should not be used) justified with three argumentation perspectives: Effectiveness of vaccination (Vaccines are not effective, because vaccinated people contracted the disease*) Consistency of vaccination (Viral vaccinations are dangerous because of the production process*) Consequences of vaccination (A child became autistic after vaccination and another child had narcolepsy*)

Note. St. = Student; * = Authentic reason included in the blog text.

Table 3. Examples of scoring of students’ responses to the YouTube video argumentation analysis task.

Score	Student response	Interpretation
0	The key point is that vaccines are complete bullshit :) (St. 134)	Source position not correctly identified (Vaccines are bullshit)
1	Vaccines protect against many diseases and deaths (St. 115)	Source position (for vaccination) correctly identified (Vaccines protect against diseases).
2	Vaccines do not cause autism and vaccines save people from bad diseases. The most important justifications were the use of visuals showing how vaccinated people can protect themselves from diseases and demonstrating the number of deaths from these diseases when not vaccinated. (St. 95)	A descriptive claim (Vaccines save people from bad diseases) justified from one argumentation perspective: Effectiveness of vaccination (Vaccines are effective, because they protect people from diseases*)
3	The main message is that vaccination is not useless. The most important justification is the survival of vaccinated children. (St. 354)	Correct conclusion (Vaccination is not useless) justified with one argumentation perspective: Effectiveness of vaccination (The most important justification is the survival of vaccinated children*)
4	The main point is that it is stupid not to vaccinate a child. Therefore, the video supports vaccines. The opinion was justified by listing many diseases against which vaccinations are used and saying that one can get the disease if not vaccinated. However, the video mentions that research exists claiming that vaccinations cause autism. However, this claim was completely refuted in the video. (St. 17)	Correct conclusion (It is stupid to not vaccinate a child) justified with two argumentation perspectives: Effectiveness of vaccination (Vaccinations protect children from many diseases*) Consequences of vaccination (Vaccinations are claimed to cause autism, which is refuted in the video*)

Note. St. = Student; * = Authentic reason included in the video.

Table 4. Variables used in the study.

Variable	N	Scoring range
Dependent skill variables
Credibility rating for the blog	403	1–3
Credibility rating for the video	403	1–3
Credibility evaluation for the blog	403	0–4
Credibility evaluation for the video	403	0–4
Identifying unbalanced argumentation in the blog	403	0 = No; 1 = Yes
Identifying unbalanced argumentation in the video	403	0 = No; 1 = Yes
Argument analysis for the blog	397	0–4
Argument analysis for the video	403	0–4
Independent variables
Grades
Philosophy	273	4–10
Psychology	337	4–10
Health education	311	4–10
Mother tongue and literature	344	4–10
Mathematics	331	4–10
History	342	4–10
Other
Reading fluency	404	0–100
Prior knowledge	404	0–3
Gender	401*

Note. * Females, n = 244 (60.6%); Males, n = 157 (39.2%).

Table 5. Level of credibility evaluation and argument analysis for the blog text and the YouTube video.

	0 points		1 point		2 points		3 points		4 points		M	SD
Skill variable	f	%	f	%	f	%	f	%	f	%
Credibility evaluation for the blog	14	3.5	55	13.6	148	36.7	127	31.5	59	14.6	2.4	1.0
Credibility evaluation for the video	35	8.7	32	7.9	205	50.9	108	26.8	23	5.7	2.1	1.0
Argument analysis for the blog	5	1.3	130	32.7	168	42.3	83	20.9	11	2.8	1.9	0.8
Argument analysis for the video	4	1.0	83	21.0	124	31.3	145	36.6	40	10.1	2.3	1.0

Figure 1. Standardized parameter estimates (significant at the.05 level) for the structural equation model explaining students’ skills in argument analysis and credibility evaluation.

Note. Ovals to the left of the grades refer to the specific factors for each grade (e.g., Phil_SF = specific factor for philosophy grade).