Students Evaluating and Corroborating Digital News

ABSTRACT

In this study, we investigate how 2,216 Swedish upper secondary school students’ performances of sourcing, evaluating evidence, and corroborating digital news relate to their background, educational orientation attitudes, and self-rated skills. We used a combined online survey and performance test to investigate students’ abilities to evaluate online news. Findings confirm and challenge previous research results about civic online reasoning. The most prominent effect on performance is the appreciation of credible news. This attitude is related to students’ abilities to source news, evaluate texts and images, and corroborate a misleading climate change website. We also found a digital civic literacy divide between students on theoretical and vocational programs with different knowledge, skills, and attitudes. Noting the democratic challenge of misinformation, we call for more research on how education can support digital civic literacy in general and specific ways.

KEYWORDS:

• Replication
• credibility
• sourcing
• fact-checking
• fake news
• media and information literacy
• disinformation
• critical thinking
• misinformation

Introduction

In a world of information disorder and infodemics, researchers and authorities call for actions against misinformation¹ (European Commision, 2018; Wardle & Derakhshan, 2017; Zarocostas, 2020). In this call, education is described as a key defense against disinformation (European Commision, 2018). Since automated fact-checking and legislation hold many limitations, researchers note that citizens, and not least young people growing up in a digital world, need updated knowledge, skills, and attitudes to use new media wisely (Carlsson, 2019; Mihailidis, 2018). The ability to read and evaluate digital information has been described as a “survival skill” for citizens in a digital world (Eshet, 2004). Today, international and national guidelines for education emphasize the importance of students’ media and information literacy in the fight against misinformation. Nevertheless, many students struggle to determine the credibility of online information (Breakstone et al., 2019; Ku et al., 2019; Nygren & Guath, 2019).

From a psychological point of view, there are many reasons why it is difficult for people to determine the credibility of online information. People may follow misleading visual cues and use flawed heuristics to assess the content’s trustworthiness (Fogg et al., 2003; Metzger & Flanagin, 2013; Sundar et al., 2007). In addition, assessments may be influenced by confirmation biases (Kahan, 2017; Nickerson, 1998). However, it has also been noted that young people’s knowledge, skills, and attitudes may support their abilities to determine the credibility of digital news (Nygren & Guath, 2019).

Considering that accurate information is a pivotal part of democracy, it is of great importance to acquire a better understanding of adolescents’ abilities to assess the credibility of online news. Not least since it is difficult to correct false and biased perceptions, even when people discover that they have been misinformed (De Keersmaecker & Roets, 2017; Lewandowsky et al., 2012). Civic online reasoning offers a prescriptive theory of how one ought to fact-check online information (McGrew et al., 2017). Civic online reasoning has been noted among professional fact-checkers (Wineburg & McGrew, 2019) and is defined as “the ability to effectively search for, evaluate, and verify social and political information online” (McGrew et al., 2018, p. 1). Based upon theories of civic online reasoning, students’ abilities to determine the credibility of online information have been assessed (Breakstone et al., 2019; McGrew et al., 2018). However, previous research have investigated how civic online reasoning relates, only to a limited extent, to students’ backgrounds and self-rated skills and attitudes (Nygren & Guath, 2019). In the current study, we investigate how 2,216 Swedish upper secondary school students’ education, self-rated skills, and attitudes are related to civic online reasoning to better understand the challenges of misinformation.

Previous Research

News consumption can support democratic engagement and create an understanding of society, but it is also associated with segregation, in terms of access to and interest in news among Swedish adolescents (Kruikemeier & Shehata, 2017; Lindell & Hovden, 2018). Similarly, research have identified a digital divide in productive use of digital media pertaining to inequalities associated with gender and class rather than access to digital technology (Hargittai, 2001; van Dijk, 2020). Access to computers and digital media does not automatically bridge this divide, since students with poor socio-economic status may spend more time online on entertainment and simple tasks than peers with better support from home (Deursen & Dijk, 2014; Hatlevik et al., 2015).

Determining whom to trust online is very complex and intellectually challenging since source critical thinking has multiple interlinked dimensions (Nygren et al., 2020; Sundar et al., 2007). The ability to successfully navigate false, biased, and credible information has been linked to cognitive abilities and flexible thinking, whereas motivated reasoning and confirmation bias are related to less efficient navigation (Flanagin et al., 2018; Pennycook & Rand, 2018). People can also be more or less aware of malicious online problems, giving them a one-sided perspective on information as being credible (Metzger et al., 2015). Naïve internet users may trust almost anyone and are easily deceived, while more informed users are more skeptical when dealing with news and issues of security and privacy (Ku et al., 2019; Vraga & Tully, 2019).

The importance of literacy and academic literacy has been underscored in previous research (Cummins & Swain, 2014; Moje, 2007; Shanahan & Shanahan, 2012). Students’ abilities to determine credibility has been linked to their disciplinary literacy, for example, their subject specific knowledge and habits of mind (Nygren & Guath, 2019; Nygren, Haglund, et al., 2019). In theory and practice, scholars have outlined different aspects of disciplinary literacy that are necessary for evaluating information in different academic subjects. This research highlights how expertise in a discipline includes content knowledge, skills, and attitudes (Nygren, Haglund, et al., 2019; Shanahan et al., 2011; Wineburg, 1991, 1998). Among experts, it has been noted that fact-checkers are particularly skilled at using digital resources to double check information (i.e., read laterally); moreover, they are able to corroborate information in more skillful ways than trained historians (Wineburg & McGrew, 2019). It has also been noted that people who are skilled at manipulating images are more adept at identifying them (Shen et al., 2019). Other research indicate that students with educational orientation in the arts are better at identifying manipulative information (Nygren & Guath, 2019).

Performance studies on teenagers’ abilities to evaluate digital news are rare. However, a growing number of studies highlight how young people struggle to determine credibility in digital environments (Breakstone et al., 2019; Ku et al., 2019; McGrew et al., 2018; Nygren & Guath, 2019). It is evident that young people may not be as skilled as they think they are at navigating online information (Enochsson, 2018; Nygren & Guath, 2019). Even with access to a great amount of information, it may be difficult for young people to determine what sources to trust and how to make use of resources like Wikipedia and navigate socio-scientific controversy in digital environments (Blikstad-Balas, 2016; Flanagin et al., 2018; Haider & Sundin, 2019; Solli, 2019). Students may struggle to find and verify relevant information online, since they confuse popularity rankings with quality, for instance, they tend to click on top-listed hits on Google (Pan et al., 2007; Sundin & Carlsson, 2016). Experts, on the other hand, often practice click restraint and corroborate information in ways that are more sophisticated (Wineburg & McGrew, 2019).

Previous explorative research indicate that the appreciation of access to credible news and a recognition of the difficulty to evaluate sources characterize a constructive mind-set when determining the credibility of digital news. Conversely, over-confidence in one’s ability in searching for and reviewing information online has been noted as indicative of the opposite (Nygren & Guath, 2019). One possibility is that self-reflection is related to science curiosity (Kahan et al., 2017), which has been linked to better ability to assess the credibility of online information. The diverging ability to assess the credibility of online news has been described as a digital divide, which, in turn, has been linked to the concept of news seekers and news avoiders (Nygren & Guath, 2019; Strömbäck et al., 2013). Young people with different attitudes toward news may or may not learn to appreciate and navigate credible news media (Lindell & Hovden, 2018; Nygren & Guath, 2019). Attitudes toward the importance of news differ among students in theoretical and vocational programs in Sweden, that is, following the news as students in vocational programs is considered as a low prestige activity, whereas the opposite may be true for students in theoretical programs (Lindell, 2018). Despite the importance of the ability to determine the credibility of news, studies investigating the link between students’ abilities and their backgrounds, education, attitudes, and self-rated skills are lacking.

Theoretical Considerations

Media and information literacy (MIL) can be conceptualized as an umbrella term that includes multiple literacies that are useful and necessary to navigate online information (Carlsson, 2019; Koltay, 2011a, 2011b). MIL can be defined as knowledge needed to access, evaluate, analyze, and create information online and includes a number of digital literacies, such as photo-visual literacy and information literacy (Aufderheide, 1993; Eshet, 2004; Hobbs, 2010; Leaning, 2019; Livingstone, 2004). Advocates of MIL underscore how all citizens need to be able to access, evaluate, and draw conclusions regarding social and political topics based upon digital news.

Abilities to combine multiple literacies have been described as transliteracy “the ability to read, write and interact across a range of platforms, tools and media” (Thomas et al., 2007). Theories of transliteracy have recently been linked to civic online reasoning, and researchers note how skilled fact-checkers possess transliterate abilities that ordinary people also need to navigate digital news (Frau-Meigs et al., 2020). Civic online reasoning is a prescriptive theory, developed by the Stanford History Education Group, stemming from empirical studies identifying how professional fact-checkers are more skilled than historians and students when evaluating online information (Wineburg & McGrew, 2019). Specifically, the theory looks at three dimensions that fact-checkers focus on when evaluating the credibility of information, namely how fact-checkers (a) identify the source of information (sourcing), (b) identify if the information is based upon reliable information (evidence), and (c) verify information by using multiple independent sources (corroboration). The three dimensions relate to questions investigating: (a) Who is behind the information? (b) What is the evidence?, and (c) What do other sources say? (McGrew et al., 2017, 2018). The importance of scrutinizing the source, evaluating the presented evidence, and corroborating claims can be linked to theories on information literacy from the 1980s (Breivik & Gee, 1989). In this study, we use the theory of civic online reasoning to better understand some of the complexities of determining the credibility of digital news.

Present Study

In light of previous research, it is imperative to investigate the question: How do students’ abilities to determine the credibility of digital news relate to their backgrounds, education, attitudes, and self-reported abilities? To our knowledge, previous research have not investigated this search- and fact-checking ability in relation to background and self-reports.

We partially replicated the relationships that were found in a small-scale exploratory study of 483 Swedish students’ civic online reasoning (Nygren & Guath, 2019) in a larger and more representative sample of Swedish adolescents, including students in vocational programs. Departing from dimensions of civic online reasoning (Wineburg & McGrew, 2019) we measured performance as (a) sourcing: awareness of source bias, (b) corroboration: corroborating information by comparing multiple sources, and (c) evaluating evidence: assessment of the use of evidence. In line with previous research and the Swedish study by Nygren and Guath (2019), testing students’ skills in relation to background, attitudes, and self-rated abilities, we hypothesized the following:

• H1: participants speaking Swedish at home will perform better on items measuring skills of sourcing. H1 can be linked to previous research highlighting how bilingual students with immigrant backgrounds may have special challenges in school performance (Cummins & Swain, 2014; Schulz et al., 2010).

• H2: the higher the participants rate the importance of access to credible news, the better they will perform on items measuring skills of evaluating evidence. H2 is associated with research highlighting the importance of science curiosity (Kahan et al., 2017) and source trust (Haider & Sundin, 2019).

• H3: participants with an educational orientation toward arts will perform better on items measuring skills of corroboration and evaluating evidence. H3 may be associated with research highlighting the importance of domain specific skills when analyzing manipulated images (Shen et al., 2019) and disciplinary literacy (Shanahan et al., 2011).

• H4: higher self-rated abilities to search and fact-check information will be associated with poorer performance on items measuring skills of evaluating evidence. H4 links to research noting how adolescents may overestimate their information literacy skills (Enochsson, 2018).

• H5: higher ratings regarding reliability of internet information will be associated with poorer performance on items measuring skills of evaluating evidence. H5 is associated with research identifying how skeptical attitudes toward online information may be productive (Vraga & Tully, 2019).

• H6: indicating girl as gender will be associated with better performance on items measuring skills of corroboration. H6 links to research highlighting how girls often outperform boys in school (Bedard & Cho, 2010; Schulz et al., 2010).

In the present study, we used test items for all three dimensions of civic online reasoning; however, we did not include all of the test items used in Nygren and Guath (2019) in order to allow for a new test item of corroboration that is more in line with what fact-checkers do when they corroborate information using digital resources (Wineburg & McGrew, 2019).² The exclusion of items was also based upon considerations about fitting the survey into the context of a large-scale citizen science project.

Method

Participants

We asked students in Sweden to fill out a survey with self-rated questions, combined with a performance test prior to a mass-experiment, with a focus on online fact-checking. In total, 2,356 participants, aged 16–19, from 33 municipalities, recruited via their teachers, filled out the survey and completed a test in the classroom using a link shared by the teacher. However, 140 participants were excluded from the analysis: 50 participants indicated a non-specified program, 54 participants did not wish to state their gender identity, and 36 identified as a gender other than a girl or a boy. The number of participants in each cohort was 1,003 in upper secondary first year; 761 in second year; and 452 in third year. Participation was voluntary, with informed consent obtained from all of the participants. In line with ethical guidelines, we did not collect any traceable data.³ All 2,356 participants agreed to participate; after excluding the above-mentioned participants, we analyzed 2,216 (1,229 girls, 987 boys), 94% of the responses.

Participants studied a wide range of programs, but primarily theoretical higher education preparatory programs such as Social Science (32%), Natural Science (18%), Economics (16%), Technology (10%), Arts (3%), and Humanities (1%). In addition, students from vocational programs participated in Child and Recreation (4%), Vehicle and Transport (3%), Building and Construction (3%), Natural Resource Use (2%), Electricity and Energy (2%), Care and Nursing Program (2%), and Business and Administration (2%) (see Table A1). The distribution of responses roughly match the national statistics at the time data were collected and students per program was: Social Science 19%; Natural Science 15%; Economics 15%; Technology 10%, Arts 7%, Humanities 1%, Child and Recreation 3%; Vehicle and Transport 4%; Building and Construction 4%; Natural Resource Use 3%; Electricity and Energy 5%; Care and Nursing 3%; and Business and Administration 3%.⁴ In the analyses, we merged the smaller upper-secondary programs with the larger groups, resulting in seven different groups, two vocational: (a) Vocational Construction Trades (n = 202); (b) Vocational Human (n = 236), and five higher-education preparatory theoretical programs: (c) Arts and Humanities (n = 96); (d) Economics (n = 353); (e) Natural Science (n = 395); (f) Social Science (n = 718); and (g) Technology (n = 216) (see also Appendix A). Different programs in Sweden attract students with different backgrounds, interests, and ambitions. Programs usually host a diverse group of students; however, vocational programs usually attract more students with lower grades and parents from the working class, while theoretical programs attract students with higher grades and middle-class parents (Forsberg, 2015).

Material and Procedure

Objective Abilities

The survey and performance test consisted of 16 questions whose contents and measured variables are presented in Table 1 below. We used test items from Nygren and Guath (2019) that were inspired by McGrew et al. (2017, 2018) and piloted in collaboration with teachers and researchers in education and psychology. In addition to the items previously used in research to test skills of sourcing and evaluating evidence, we designed a test item to measure corroboration, with inspiration from McGrew et al. (2018) prompting students to corroborate a biased website. The site that we directed students to (www.klimatupplysningen.se) has been noted as deceptive pseudoscience by researchers (Häggström, 2008) and described as climate change denial by Swedish public service (SVT) and other established news media. In 2010, it was given the award of “Misleader of the Year” (Folkbildning, 2010).⁵ We piloted this item with a total of 173 students to safeguard validity and reliability. In the classrooms, we collected feedback from pilot-participants; thereafter, we analyzed their open-ended justifications for their responses to make sure that they made sense of the items. We found that students could understand the item and that they could successfully corroborate the item.

Table 1. Overview of questions and measured constructs in the online survey. The measured constructs refer to abilities that have been identified as important for the detecting of false news online. The questions are referred to in the text with the names in parentheses.

Measured variable	Question
Background variables	Gender
	Cohort
	Upper secondary school program
	Language at home (foreign language/Swedish)
Self-reported categories	Items in self-report
Self-rated abilities	Source criticism on the Internet (fact-checking ability)
	Finding information on the Internet (search ability)
Information credibility	Importance of news credibility (credibility importance)
	Reliability of information on the Internet (internet info reliability)
Categories of civic online reasoning	Test-items
Detecting sponsored material (sourcing)	Screenshot from evening paper 1 (Aftonbladet)
	Screenshot from evening paper 2 (Expressen)
Scrutinizing comments and images (evidence)	Manipulated photograph of a smoking girl (smoking)
Verify information laterally (corroboration)	Misleading information about climate change (climate change)

The items measuring sourcing – students’ abilities to separate ads from news – were two screenshots from two popular evening papers (www.expressen.se and www.aftonbladet.se), with a complicated mix of ads and news. The item measuring students’ abilities to evaluate evidence was a manipulated photo (by Kai Bastard) showing a smoking girl with black veins.⁶ The item measuring corroboration was a website about climate change, arguing that there is no evidence for climate change, and illustrating this with a histogram with a flat regression line. Instructions prompted students to visit the website and corroborate the information online.

Attitudes and Self-rated Abilities

The self-rated skills and attitudes questions were the same as those used in Nygren and Guath (2019), which were designed to capture basic attitudes that are part of digital literacy and civic online reasoning. Questions measure two attitudes: (a) “internet info reliability” and (b) “credibility importance”. Internet info reliability is intended to measure the participants’ general attitude regarding the reliability of online information. The attitude is related to the link between productive critical thinking, news habits, and skepticism about online information (Ku et al., 2019; Vraga & Tully, 2019). Additionally, the attitude may be connected to cognitive abilities and flexible thinking related to peoples’ abilities to navigate online information (Flanagin et al., 2018; Pennycook & Rand, 2018). Credibility importance is intended to measure the importance the participant attaches to trusting credible sources and scientific evidence. Knowing what sources to trust has been noted as central in online navigation and a key aspect of infrastructural meaning-making (Haider & Sundin, 2019). This question also taps into the concept of science curiosity (Kahan et al., 2017) and news seekers (Strömbäck et al., 2013), characterizing people who seek information with a curious mindset directed toward credible information and news. People with science curiosity have been described as less vulnerable to partisan polarization. The attitudinal questions were also measured with a 5-point scale. The question about internet info reliability was phrased as: “How much of the information on the internet that you consume do you regard as reliable?” and the question about credibility importance was phrased as: “How important is it for you to consume news that is credible?”

Finally, also in line with Nygren and Guath (2019), we measured two self-rated skills: (a) fact-checking ability and (b) search ability. Fact-checking ability intends to measure people’s self-rated ability to determine credibility, while search ability intends to measure their self-rated ability to find credible information online and corroborate information. Previous results (Nygren & Guath, 2019) indicate that it may also measure overconfidence (Kruger & Dunning, 1999), in the sense that people with poor skills in an area often tend to overestimate their actual skill. The self-rated skills were all measured with a 5-point scale. The question about source criticism was phrased as: “How good do you think you are at being source critical about the information you find on the internet?” The question about fact-checking ability was phrased as: “How good are you at finding the information you are looking for on the internet?”

The survey also included a number of background variables: (a) gender, (b) program of study, and (c) language spoken at home. An overview of the questions and measured constructs is provided in Table 1.

Results

Analysis of Results

In order to investigate how the civic online reasoning skills were related to performance, we made logistic regressions with fact-checking ability, search-ability, gender, upper secondary program, language at home, internet info reliability, and credibility importance as independent variables.

The dependent variable in logistic regressions is categorical (here, correct/incorrect), and the coefficients describe the log odds for a correct answer, with a positive coefficient denoting an increased log odds for a correct answer and vice versa. For the items measuring sourcing skills (Aftonbladet and Expressen), we coded answers as responses to multiple choice questions. All selected choices had to be correct or else the answer was coded as incorrect. This coding was in line with a recent study by Nygren et al. (2020) measuring the same abilities. We used a function that selected the model with the lowest Akaike information criterion (AIC), adding each predictor sequentially (R Core Team, 2019). Here, we only report the significant results in the text; for a full specification of the models (including model fit), we refer to Appendix B, Tables B1–B4.

Overview

All self-rated abilities were on a 5-point scale. Beginning with self-rated abilities, the participants considered themselves as being quite apt at source criticism on the internet (fact-checking: M = 3.7, SD = 0.8) and even better at finding information on the internet (search ability: M = 4.1, SD = 0.7). Participants considered it important to consume credible news (M = 4.1, SD = 1.0), and they considered information on the internet to be slightly more reliable than average (M = 2.9, SD = .7).

The self-rated abilities were somewhat misaligned with the objective abilities, because sourcing was 25% correct on Aftonbladet and 11% on Expressen, and only 3% answered both sourcing items correctly. In addition, 66% answered correctly on smoking (measuring evidence), and 49% answered correctly on climate change (measuring corroboration).

Sourcing: Detecting Sponsored Material

For Expressen, there was a larger log odds for a correct answer [b = 0.68, z = 2.30, p = .022] when attending Arts and Humanities programs compared with baseline (Social Science).⁷

For Aftonbladet, there were four significant effects: for a unit increase on the rating of fact-checking ability, there was a larger log odds for a correct answer [b = 0.22, z = 2.94, p = .003]; for a unit increase on the rating of credibility importance there was a larger log odds for a correct answer [b = 0.28, z = 4.47, p < .001]; for upper secondary school program, there was a larger log odds for a correct answer when attending Natural Science [b = 0.41, z = 2.87, p = .004] and Technology programs [b = 0.58, z = 3.20, p = .001] compared with baseline Social Science. Moreover, there was a smaller log odds for a correct answer when attending vocational Construction and Trades programs [b = − 0.61, z = − 2.58, p = .01] compared with baseline Social Sciences; and when speaking Swedish at home, there was a larger log odds for a correct answer [b = 0.43, z = 3.19, p = .001].

Evidence: Scrutinizing Comments and Images

For correctly evaluating evidence, there were six significant predictors: (a) for a unit increase on the rating of search ability, there was a larger log odds for a correct answer [b = 0.22, z = 3.08, p = .002]; (b) for a unit increase in the rating of credibility importance, there was a larger log odds for a correct answer [b = 0.17, z = 3.20 p = .001]; (c) when indicating gender “boy,” there was a larger log odds for a correct answer [b = 0.50, z = 4.73, p < .001] compared with baseline “girl”; (d) for a unit increase on the rating of internet info reliability, there was a smaller log odds for a correct answer [b = − 0.20, z = − 3.04, p = .002]; (e) for upper secondary school program, there was a smaller log odds for a correct answer when attending vocational Human programs [b = − 0.42, z = − 2.25, p = .024] or the theoretical Economic program [b = − 0.36, z = − 2.33, p = .020] compared with baseline Social Science, and there was a larger log odds for a correct answer when attending Natural Science program [b = 0.87, z = 6.45, p < .001] or Technology program [b = 0.49, z = 2.84, p = .004] compared with Social Science; (f) and when speaking Swedish at home, there was a larger log odds for a correct answer [b = 0.39, z = 3.21, p = .001].

Corroboration: Verify Information Laterally

For corroborating the text on climate change correctly, there were three significant predictors: (a) for a unit increase on credibility importance, there was a larger log odds [b = − 0.14, z = 2.78, p = .005] for a correct answer; (b) for a unit increase on internet info reliability there was a smaller log odds for a correct answer [b = −0.22, z = −3.62, p < .001]; and (c) for attending the Natural Science program (compared with baseline Social Science), there was larger log odds [b = 0.44, z = 3.35, p < .001] for a correct answer.

Discussion

Results from this study measuring adolescents’ civic online reasoning skills in relation to background variables, attitudes, and self-rated abilities showed that valuing credible news, studying at the Natural Science program, and speaking Swedish at home were often related to better performance. Rating information on the internet as reliable or studying in a vocational program were related to poorer performance. In addition, the current study provides new insights on how background, education, and self-rated abilities affect students’ performance when corroborating misleading online information about climate change. We find that determining the credibility of this website was a challenge for a majority of the students; however, students with subject specific knowledge in the Natural Science program seemed better at navigating this online information. In addition, students who appreciated reliable news were better at debunking the climate change denial. The results partly corroborate previous results in the smaller explorative study by Nygren and Guath (2019). We discuss the results in relation to our hypotheses and conclude with recommendation for future research.

Language at Home

Speaking Swedish at home had a positive effect on performance on sourcing (Aftonbladet), in line with H1 and the previous study by Nygren and Guath (2019). In addition, speaking Swedish at home was also related to better performance on the item testing skills to evaluate evidence (smoking). We speculate that the relationship is linked to basic literacy (which, in turn, is connected to information literacy), literacy skills among bilingual students (Cummins & Swain, 2014), and news habits among different groups of teenagers (Lindell, 2018; Lindell & Hovden, 2018). We do not find this effect on corroboration (climate change denial), which highlights that other aspects are important when navigating online – like subject specific knowledge and appreciating reliable news. Further investigation is needed in order to provide concrete instructions on how this can be addressed in multicultural classrooms.

Credibility Importance

Results underscore the importance of the appreciation of credible news (credibility importance). In line with H2, we find that importance of credibility relates to better performance on items testing skills of evaluating evidence. This result echoes findings in previous research and expands the explanatory value of credibility importance to also include skills of sourcing and corroboration. We find that students with higher ratings on the importance of access to credible news performed better on all three dimensions of civic online reasoning – sourcing, evidence, and corroboration. A possible explanation for the association between credibility importance and performance may be that valuing credible news is linked to science curiosity (Kahan et al., 2017), helping students to evaluate information with a better focus on facts beyond motivated reasoning.

In line with previous research, we also speculate that credibility importance may be linked to students’ trust in sources (Haider & Sundin, 2019) and habits of following public service TV and listening to the radio (Nygren & Guath, 2019). It is also possible that there is a link between regarding access to credible news as important and media habits. This is supported by research investigating the link between social media habits and source critical ability (Ku et al., 2019). We further speculate that this group may contain news seekers (Strömbäck et al., 2013), who find credible news in public service. Those students may be part of a culture where news is perceived as high status (e.g., Lindell, 2018; Lindell & Hovden, 2018). Our findings emphasize the importance of further investigation of what constitutes credibility importance and how the attitudes can be stimulated in education.

Educational Orientation

Results also point at the importance of educational orientation for the performance. Partly in line with H3, we find that students enrolled in the Arts programs performed well on items testing skills of sourcing and evaluating evidence. We find that students in programs for Technology, Art and Humanities performed well on items testing abilities of sourcing and evaluation of evidence, but not on the item testing abilities to corroborate online information.

In contrast to H3, we find that students in the Natural Science program performed significantly better than other students on the item measuring corroboration (climate change denial). They may have more knowledge about this issue than other students, helping them when reading laterally. Theories of disciplinary literacy predict that students in Natural Sciences have subject specific knowledge that makes it easier for them to search for credible information and determine the credibility of information relating to natural sciences (Nygren, Haglund, et al., 2019; T. Shanahan & Shanahan, 2012). Students with an orientation toward the Natural Sciences also performed well on other items measuring sourcing and evidence, highlighting possibly a more general ability.

The fact that Technology students were able to debunk a manipulated image may be linked to their knowledge about digital image manipulation. This interpretation is supported by previous research, highlighting how debunking manipulated images may relate to technological expertise (Shen et al., 2019). We also speculate that the abilities of students in the Arts and Humanities programs to debunk misleading texts and images may be related to disciplinary literacy.

We also find that students in vocational programs and students in the theoretical Economy program performed significantly worse on items testing skills to identify the source and evaluate evidence. This may indicate a digital divide (van Dijk, 2020), further discussed below.

Search Ability and Fact-checking Ability

Previous findings regarding over-confidence problems among teenagers, noted by Nygren and Guath (2019), were not supported in this replication. Contrary to our hypothesis (H4), we found that students claiming to be good at fact-checking performed well on items testing their abilities to identify the source, and those claiming to be good at searching for information performed better when evaluating evidence. The reason for these results may be that we used a larger and more representative sample of participating students; notably, we included students with vocational education. The updated curriculum in Sweden, emphasizing the importance of digital literacy, may also have influenced the results (Regeringen, 2017). At the time of the survey, there was also a lively discussion about problems with “fake news” and people’s problems with determining credibility.

Our findings also relate to previous research highlighting how self-reflection and skills have complex links. Interestingly, different self-rated abilities were related to different abilities; fact-checking ability had a positive effect on sourcing (Aftonbladet), whereas search ability had a positive effect on evidence (smoking). Fact-checking ability measures the self-rated ability for source criticism, which is a central skill when sourcing, whereas search-ability measures the self-rated ability to find and consume credible information. Nygren and Guath (2019) did not identify this relation in their study. The finding is positive in two respects: (a) it gives an indication that the items measure what they intend to measure and (b) it shows that the students possess a degree of self-knowledge. In related research, it was found that students who are good at debunking the smoking image often do this by comparing the content of the image with what they know from other sources of information (Nygren et al., 2020), which may be related to search ability.

It is also noteworthy that we did not find any effects of the self-rated skills of fact-checking or search ability on corroboration, which may indicate that students who are able to corroborate information like fact-checkers are not aware of their skills to search and evaluate online information. Another interpretation is that the effects of credibility importance, internet info reliability, and education on this item are far greater, hence overshadowing any effects of self-rated abilities.

Internet Info Reliability

The present results are in line with H5 showing that higher ratings of reliability of internet information are associated with poorer performance on items measuring skills of evaluating evidence. Findings echo those in the previous explorative study regarding the negative relationship between performance and ratings of internet information as reliable (Nygren & Guath, 2019). Beyond previous research, we also find a negative relationship between trusting online information and abilities to corroborate misleading information about climate change. We speculate that the negative relationship between trusting internet information and skills of determining credibility may be explained by the fact that naïve internet users are easier to manipulate than those who are vigilant about misinformation. Previous research have noted how naïve attitudes may be problematic when reading news online (Vraga & Tully, 2019), which is confirmed by research indicating a relationship between critical ability and skepticism about news algorithms (Ku et al., 2019).

Another possible explanation comes from research emphasizing the challenge of combatting confirmation bias when evaluating credibility online (Metzger & Flanagin, 2015). Participants may perceive one-sided biased arguments as more credible, since they confirm their current views, thus avoiding cognitive dissonance, and consequently falling victim to confirmation bias (Metzger et al., 2015). This, in turn, may be attributed to a lack of flexible thinking among participants rating internet information as credible; they are less able to see beyond the face value and their own biases (Flanagin et al., 2018).

However, previous research also show that high credibility ratings of internet information can be linked to self-reported verification behavior. People who report that they often corroborate information also report that they trust online information, which is in contrast to what we found in this study (Flanagin & Metzger, 2000). We believe that it is possible that this difference can be explained by the fact that we test abilities and relate them to self-reported measures, while previous research only used self-reported measurements. Evidently, more research is needed on why people in general trust and distrust online information, and how this relates to their actual abilities to navigate credible, biased, and false information.

Gender

In contrast to our hypothesis (H6), we did not find that girls were better at corroborating information. The only effect of gender was on evaluating evidence (smoking) where girls more than boys were fooled by this manipulative image and text. Even if girls often outperform boys (Bedard & Cho, 2010; Schulz et al., 2010), we found the opposite. We speculate that the result may be related to the fact that the smoker in the image is a girl, attracting compassion rather than critical distance among female participants. This is confirmed by research showing that images evoke more compassion in women than in men (Groen et al., 2013; Mercadillo et al., 2011).

We also speculate that this can be explained by habits of navigating manipulated body images. The newsfeeds of Swedish teenage girls are filled with celebrity gossip and misleading images of beauty (Nygren, Brounéus, et al., 2019). In addition, girls in Sweden spend more time on social media than boys, who spend more time on gaming (Statens medieråd, 2019). Images in social media may impact young women, giving them a biased view of the body (Holland & Tiggemann, 2016; Marengo et al., 2018; Walker et al., 2019). We speculate that the results can be related to previous research highlighting how teenage girls may have poor mental health literacy in a digital world of selfies, plastic surgery, and manipulated images (Lonergan et al., 2019).

Researchers suggest that media literacy training may be useful to better navigate in social media with images of manipulated bodies (Khazir et al., 2016; McLean et al., 2017). Yet another interpretation is that the results are related to participants’ smoking habits, but we did not ask about this in the survey. Further research is needed to better understand this gender difference and how this relates to media literacy.

Limitations

Our findings highlight the complexity of determining credibility. However, the study has important limitations since we only tested a few aspects of this complexity with a limited number of test-items. In the performance test, we used four items that were intended to measure the three facets of civic online reasoning. Ideally, we would have used multiple items for measuring the different skills; however, since this was part of a mass experiment, the time frame did not allow us to include more items. This has implications for the validity of the performance test, for instance, does it capture the facets of the civic online reasoning and is it predictive of performance on each skill it intends to measure? The validity weaknesses are also reflected in the reliability of the test – do similar items within a facet of civic online reasoning measure the same construct? Our results on sourcing, where we included two items, indicate that they do not, since performance was predicted by different variables. Future research should include more items in each category, and preferably map the attitudes and self-rated measures with established scales on each topic. We are currently gathering data, with the intention of mapping our self-rated variables with validated questionnaires. This would make it possible to tie them to validated constructs and, consequently, say something more substantial about the predictors. Another limitation of this study is the fact that interested teachers signed up to participate and students filled in questionnaires and tests in a classroom environment, and their vigilance regarding manipulations may therefore have been higher than otherwise when encountering online information. Performance may also have been affected by the fact that this was a low stakes test that did not affect their grades. To what extent our findings translates into teenagers’ personal use of online information on a national scale is beyond the scope of this study. What we do see are some important challenges in a school setting that are greater in some groups of students. We find what seems to be a digital civic literacy divide.

Conclusion: A Digital Civic Literacy Divide

In this study, we tap into the complexity of determining credibility in digital environments. Departing from theories of transliteracy and civic online reasoning, we find a digital divide between young people regarding their knowledge, skills, and attitudes. This divide may, in turn, be described as a divide of digital civic literacy. We define digital civic literacy as knowledge, skills, and attitudes necessary to navigate multimodal online information of importance to citizens in critical and productive ways. Our results suggest that students with a high digital civic literacy have subject specific knowledge and disciplinary knowledge closely related to issues in the news; furthermore, they know where to find credible information and know that texts and images may be manipulated. Moreover, they are able to determine the credibility of news using reasoning skills to evaluate information and skills to navigate online in productive ways. In addition, they have attitudes appreciating credible news and a vigilant attitude toward internet information in general. Finally, they display self-reflection in their attitudes regarding their abilities to fact-check and search information.

In contrast, students with low digital civic literacy do not appreciate credible news to the same extent, which may be attributed to a naïve trust in online information. The fact that students with lower digital civic literacy often attend vocational programs and do not speak Swedish at home highlights how the digital divide is a social and educational divide that needs to be addressed in future research and education. We also need a better understanding of how education may support students to appreciate credible news, learn to be vigilant toward online information, and be self-reflective about fact-checking.

Acknowledgements

We are truly grateful to all the students accepting to participate in this study as well as their teachers who helped by distributing the survey. We would also like to direct a special thanks to Jenny Wiksten Folkeryd, Ebba Elwin, Fredrik Brounéus, Kerstin Ekholm, Maria Lindberg, and members of the Stanford History Education Group for their valuable input and support in the process.

Disclosure Statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by VINNOVA: [Grant Number 2018-01279].

Notes

1 In this article, we define misinformation as inaccurate, manipulative, or false information, including disinformation, which is deliberately designed to mislead people.

2 See more details about this test item below in the Methods section.

3 http://www.codex.vr.se/forskninghumsam.shtml.

4 SiRiS, National Statistics from Swedish National Agency for Education, https://www.skolverket.se/skolutveckling/statistik/sok-statistik-om-forskola-skola-och-vuxenutbildning [accessed 2019 01 12].

5 Thus, a lateral reading of this site should make students skeptical. Previous items testing skills of corroboration have prompted students to compare two competing sources of information, but not prompting them to use digital resources to double-check information (Nygren & Guath, 2019). Noting the importance of using digital resources in lateral reading (Wineburg & McGrew, 2019), we chose to replace the test items used in Nygren and Guath (2019) with this item, emphasizing the combination of searching and evaluating information to corroborate claims.

6 Image source: https://www.vice.com/en_uk/article/yp54bw/kai-bastard-photo-manipulations.

7 The results are not affected by the category that is chosen as baseline. In our case, it was automatically coded as baseline in the software (R) which we used to analyze the data.

Appendices

Appendix A

Table A1. List of categories based upon programs of participants (n = 2216).

Program categories	Program	N	Proportion
Vocational Construction Trades	BA Bygg- och anläggningsprogrammet [Building and Construction]	70	.03
Vocational Human	BF Barn- och fritidsprogrammet [Child and Recreation]	94	.04
Vocational Construction Trades	EE El- och energiprogrammet [Electricity and Energy]	50	.02
Economics	EK Ekonomiprogrammet [Economics]	353	.16
Arts & Humanities	ES Estetiska programmet [Arts]	74	.03
Vocational Construction Trades	FT Fordon & transport [Vehicle and Transport]	72	.03
Vocational Human	HA Handel- och administrationsprogrammet [Business and Administration]	45	.02
Arts & Humanities	HU Humanistiska programmet [Humanities]	22	.01
Vocational Human	HV Hantverksprogrammet [Craft]	3	.001
Vocational Construction Trades	IN Industritekniska programmet [Industrial Technology]	2	.0009
Natural Science	NA Naturvetenskapsprogrammet [Natural Science]	395	.18
Vocational Human	NB Naturbruksprogrammet [Natural Resource Use]	54	.02
Vocational Human	RL Restaurang- och livsmedelsprogrammet [Restaurant and Food]	6	.003
Social Science	SA Samhällsvetenskapsprogrammet [Social Science]	718	.32
Technology	TE Teknikprogrammet [Technology]	216	.10
Vocational Construction Trades	VF VVS- och fastighetsprogrammet [Plumbing and Real Estate]	8	.004
Vocational Human	VO Vård- och omsorgsprogrammet [Care and Nursing]	34	.02

Appendix B

Table B1. Estimates of Best Fitting Logistic Regression Model for Correct/Incorrect Coded Multiple Choice Answers on Expressen with Coefficients Denoting the Log Odds of Answering Correct for a Unit Increase of the Ordinal Predictors and the Expected Log Odds for Each Category Compared with the Baseline Category for the Categorial Predictors.

Predictor	Estimate	Std.Error	z-value	p-value
(Intercept)	− 3.17	0.65	− 4.90	<.001 ***
Gender: Boy	− 0.14	0.16	− 0.85	.40
Fact-checking ability	0.10	0.10	1.07	.29
Credibility importance	0.06	0.08	0.69	.49
Internet info reliability	0.09	0.10	0.96	.34
Program: Vocational construction trades	0.14	0.29	0.46	.65
Program: Vocational human	0.006	0.26	0.02	.98
Program: Arts & Humanities	0.68	0.30	2.30	.022 *
Program: Natural science	−0.06	0.22	− 0.28	.78
Program: Economics	0.28	0.21	1.35	.18
Program: Technology	0.06	0.28	0.20	.84
Language: Swedish	− 0.20	0.17	− 1.16	.24
Search ability	0.07	0.11	0.62	.53

Model fit (compared with null model): χ²(12)   =13.01, p < .001

Significance codes: * < .05, ** < .01, *** < .001.

Table B2. Estimates of Best Fitting Logistic Regression Model for Correct/Incorrect Coded Multiple Choice Answers on Aftonbladet, with Coefficients Denoting the Log Odds of Answering Correct for a Unit Increase of the Ordinal Predictors and the Expected Log Odds for Each Category Compared with the Baseline Category for the Categorial Predictors.

	Estimate	Std.Error	z-value	Pr(>|z|)
(Intercept)	− 3.78	0.49	− 7.70	< .001 ***
Gender: Boy	0.17	0.11	1.47	.14
Fact-checking ability	0.22	0.07	2.94	.003 **
Credibility importance	0.27	0.06	4.47	< .001 ***
Internet info reliability	− 0.04	0.07	− 0.51	.61
Program: Vocational construction trades	− 0.61	0.24	− 2.58	.01 **
Program: Vocation human	− 0.21	0.19	− 1.09	.27
Program: Arts & Humanities	− 0.13	0.26	− 0.48	.63
Program: Natural sciences	0.41	0.14	2.87	.004 **
Program: Economics	− 0.31	0.17	− 1.87	.06
Program: Technology	0.58	0.18	3.20	.001 **
Language: Swedish	0.43	0.13	3.19	.001 **
Search ability	0.09	0.08	1.17	.24

Model fit (compared with null model): χ²(12) = 111.23, p < .001.

Significance codes: * < .05, ** < .01, *** < .001.

Table B3. Estimates of Best Fitting Logistic Regression Model for Correct/Incorrect on Smoking, with Coefficients Denoting the Log Odds of Answering Correct for a Unit Increase of the Ordinal Predictors and the Expected Log Odds for Each Category Compared with the Baseline Category for the Categorial Predictors.

Predictor	Estimate	Std.Error	z-value	p-value
(Intercept)	− 2.34	0.41	− 5.65	<.001 ***
Gender: Boy	0.50	0.11	4.73	<.001 ***
Credibility importance	0.17	0.05	3.20	.001 **
Internet info reliability	− 0.20	0.07	− 3.04	.002 **
Program: Vocational construction trades	− 0.18	0.19	−0.94	.35
Program: Vocational human	− 0.42	0.19	− 2.25	.024 *
Program: Arts & Humanities	0.31	0.23	1.33	.19
Program: Natural sciences	0.87	0.14	6.45	<.001 ***
Program: Economics	− 0.36	0.16	− 2.33	.020 *
Program: Technology	0.49	0.17	2.84	.004 **
Language: Swedish	0.39	0.12	3.21	.001 **
Search ability	0.22	0.07	3.08	.002 **

Model fit (compared with null model): χ²(11) = 165.72, p < .001.

Significance codes: * < .05, ** < .01, *** < .001.

Table B4. Estimates of Best Fitting Logistic Regression Model for Correct/Incorrect on Climate Change, with Coefficients Denoting the Log Odds of Answering Correct for a Unit Increase of the Ordinal Predictors and the Expected Log Odds for Each Category Compared with the Baseline Category for the Categorial Predictors.

Predictor	Estimate	Std.Error	z-value	p-value
(Intercept)	− 0.38	0.39	− 0.98	.33
Gender: Boy	− 0.17	0.10	− 1.72	.085
Fact-checking ability	− 0.05	0.06	− 0.83	.41
Credibility importance	0.14	0.05	2.78	.005 **
Internet info reliability	− 0.22	0.06	− 3.62	<.001 ***
Program: Vocational construction trades	0.12	0.18	0.66	.51
Program: Vocational human	− 0.17	0.16	− 1.07	.28
Program: Arts & Humanities	− 0.26	0.22	− 1.14	.26
Program: Natural sciences	0.44	0.13	3.35	<.001 ***
Program: Economics	− 0.19	0.14	− 1.41	.16
Program: Technology	− 0.15	0.17	− 0.88	.38
Language: Swedish	0.04	0.11	0.35	.72
Search ability	0.05	0.06	0.77	.44

Model fit (compared with null model): χ²(12) = 50.86, p < .001.

Significance codes: * < .05, ** < .01, *** < .001.