Exploring the effects of physics explainer videos and written explanations on declarative knowledge and the illusion of understanding

ABSTRACT

Studies have shown the potential of explainer videos. An alternative is a written explanation, as found in science textbooks. Prior research suggests that instructional explanations sometimes lead to the belief that a topic has been fully understood, even though that is, not the case. This ‘illusion of understanding’ may be affected by the medium of the explanation. Also, prior studies comparing the achievement from explainer videos and writen explanations come to ambiguous results. In the present experimental study (video group: n_V = 78, written explanation group n_W= 72), we compared the effects of an explainer video introducing the concept of force to a written explanation containing the script of the video, as though it is a page from a textbook. Both groups achieved comparable degrees of declarative knowledge, however, the written explanation video group had a significantly higher belief of understanding (partial η²= 0.043) that did not correspond with their actual learning progress. Consequences of this may include lower cognitive activation and less motivation in science classrooms if learning environments exclude further learning tasks that allow for a more realistic picture of understanding. That might suggest that it is sometimes potentially harmful to leave physics learners to their own devices with instructional material.

KEYWORDS:

• Explainer video
• science textbooks
• illusion of understanding

Introduction

In both informal and formal science education, explainer videos have gained immense popularity in recent years. Students watch explainer videos online, for example, to prepare for exams or simply for entertainment (Wolf & Kratzer, 2015; Wolf & Kulgemeyer, 2016). Formats, such as the flipped classroom, have further fostered the use of explainer videos. Many teachers now produce explainer videos for students who watch them at home and work on related learning tasks in the classroom afterwards (Wagner & Urhahne, 2021). Lately, the various purposes of explainer videos have led to numerous studies in science education and psychology on their effectiveness (e.g. Findeisen et al., 2019; Kulgemeyer, 2018a; Mayer, 2021). Such studies have addressed not only the potential of explainer videos for science teaching but also their limitations, including a potential shift to non-constructivist learning environments (Kulgemeyer, 2018a).

Kulgemeyer and Wittwer (2022) showed that, similar to instructional explanations, some physics explainer videos presenting common misconceptions as explanations fostered an ‘illusion of understanding’ – the false belief that an explanation was easy, a topic has been thoroughly understood and requires no further instruction. The illusion of understanding has potentially harmful consequences for further teaching. It might lead to perceiving further instruction as redundant and irrelevant (similar to Acuna et al., 2011) and, ultimately, a lower cognitive activation in science teaching (Kulgemeyer & Wittwer, 2022). Psychological research suggests that pictures may influence one’s sense of how much has been understood (Salomon, 1984), thus leading to an inaccurate estimation of one’s own learning progress. Students sometimes assume themselves to be performing better in a knowledge test while learning with video compared to print media (Dahan Golan et al., 2018). The ‘shallowing hypothesis’ suggests that the medium is important to the perception of the explanation. For example, learners are used to superficially interacting with online media regularly, aiming for quick rewards such as a ‘thumbs up’ (Salmerón et al., 2020). This may overshadow how they interact with the explainer videos in the context of learning. Explainer videos, involving complex principles (as in physics), may be more likely to lead to an illusion of understanding than written explanations. However, it is also possible that written explanations, especially in the form of textbook entries, are more likely to lead to an illusion of understanding. A (reverse) shallowing hypothesis may result in textbook entries and written explanations appearing to be more reliable and formal to learners. Supposedly, learners’ regular interactions with textbook entries are less superficial as the latter appear mostly in a formal learning context. Thus, they may even be convinced that they have learned more from textbook entries than they actually did, again resulting in an illusion of understanding. Regarding achievement, Kulgemeyer (2018a) suggests that explainer videos primarily foster the achievement of declarative knowledge. Studies, however, come to ambiguous results. Lee and List (2019) found an advantage of learning from text, List and Ballenger (2019) did not find a clear difference.

Since explainer videos have become an increasingly established medium in science teaching, their limitations should be examined as closely as their potential, and the present study contributes to this goal precisely. In the present experimental study, we address the question of whether physics explainer videos, compared to written explanations (as present in science textbooks), are more likely to result in the ‘illusion of understanding’. Also, we explore the effects on the achievement of declarative knowledge for the case of a short introduction into the concept of force. The comparison between written explanations and explainer videos is noteworthy for science education as it may help teachers choose between an explainer video and a textbook, when the need arises. Consequences for science teaching are also discussed.

Literature review

Learning science with explainer videos

Explainer videos are usually short videos, providing comprehensible explanations of principles (Findeisen et al., 2019; Kulgemeyer, 2018a). According to Findeisen et al. (2019), explainer videos should be limited to a length of 6 min, taking into consideration the limitations of one’s working memory. In recent studies, they have alternatively been referred to as, e.g. educational videos (Brame, 2016), or instructional videos (Schroeder & Traxler, 2017).

Wolf and Kulgemeyer (2016) argue that explainer videos can contribute to science education in many different ways depending on who the explainer and explainee are (science teacher/online source versus science learner and science teacher versus science learner, respectively). Learning from videos is a common sight in (flipped) science classrooms. Wolf and Kulgemeyer (2016), however, remind us that tasks in which students produce their own videos also have high educational value. They are potential learning opportunities for both science content knowledge and science communication skills. Wolf and Kulgemeyer (2016) further argue that analysing explainer videos can be useful for science teachers (e.g. to learn alternative explaining approaches) and for learners (e.g. to see alternatives to their teachers’ explanations). In an analysis of the literature since 1957, Pekdag and Le Marechal (2010) highlight the potential benefits of including videos in learning environments in chemistry education. They argue that explainer videos might even replace written explanations in textbooks (Pekdag & Le Marechal, 2010, 14) – a question the present study addresses.

Over the years, numerous studies have examined the videos’ effects on students’ achievements. Hartsell and Yuen (2006) remind us that online explainer videos have the potential to deconstruct teachers’ monopoly of knowledge presentation by offering alternative explanations that are easily accessible. Other studies provide evidence for interactivity being key to learning (e.g. Delen et al., 2014; Hasler et al., 2007).

One of the main research topics in this area involves the design principles underlying explainer videos, and the impact of the same on the videos’ effectiveness. These design principles have mostly been derived from multimedia learning (Findeisen et al., 2019; Mayer, 2021) and science education research (Kulgemeyer, 2018b).

For example, Brame (2016) postulates guidelines based on the cognitive load theory, including ways to reduce the extraneous cognitive load by ‘weeding’ (erasing unnecessary information). In addition, Schroeder and Traxler (2017) suggest that reducing distractions positively influences achievement. This finding mirrors the one from studies on instructional explanations, which highlight the benefits of focusing solely on the concept instead of on irrelevant details (‘minimal explanations’) (Anderson et al., 1995).

Hoogerheide et al. (2016) observed that, compared to peer explanations, adults as explainers in videos appear more competent, thus influencing the learning outcome. Seidel et al. (2013) suggest that the method of introducing the explained principle and, afterwards, illustrating it with examples might be superior fruitful in achieving content knowledge (rule-example strategy). The main quality criterion, however, is adapting to prior knowledge (Wittwer & Renkl, 2008). The tools required to reach such levels of adaptation have been empirically validated for science education in particular, and may even be domain-specific for science (Kulgemeyer & Schecker, 2009, 2013): (1) the language level, (2) the level of mathematisation, (3) representation forms and demonstrations, and (4) examples, models, and analogies. Adaptation cannot easily be achieved through a medium such as explainer videos (Findeisen et al., 2019; Kulgemeyer, 2018a; Kulgemeyer & Peters, 2016) because, other than instructional explanations as provided by teachers, they cannot include a diagnosis of the students’ understanding, followed by a more elaborate process of adaptation. Integrating interactive elements (Merkt et al., 2011) is, therefore, important. In addition, the integration of explainer videos into a learning process, for instance, by incorporating learning tasks, is a criterion for success (Altmann & Nückles, 2017; Webb et al., 2006). Lloyd and Robertson (2012) suggest that procedural knowledge may be better gained from explainer videos as compared to print media. For learning physics, explainer videos are effective in imparting declarative knowledge: The explained principle can be used after the video to explain the examples that are shown in the video. To achieve flexible conceptual knowledge, it might be necessary to incorporate learning tasks that require the application of the explained principle to unknown examples (Kulgemeyer, 2018b). In addition, explainer videos are most effective for learners with low prior knowledge; for learners with higher prior knowledge, self-explanatory attempts work better (Kulgemeyer, 2018b; Wittwer & Renkl, 2008). Acuna et al. (2011) found that learners with a low prior knowledge benefitted from instructional explanations while learners with a high knowledge did not. Wittwer and Renkl (2008) highlighted that instructional explanations were likely to fail for learners with high prior knowledge. While they have a role when introducing a new concept to novice learners, they are not useful in a more advanced stage of a teaching unit (Kulgemeyer, 2018b). Thus, for the present study, we decided to focus on learners with a low prior knowledge because they are the target group of explainer videos.

The ‘illusion of understanding’ and learning from text and video

The ‘illusion of understanding’ refers to the mistaken belief that one has understood a topic and the instruction was easy, which is both actually not the case (Chi et al., 1994; Wittwer & Renkl, 2008). Therefore, it is a special case of a belief in understanding with a difference between this assumed degree of understanding and objective comprehension but also the impression that learning material is easier than it actually is. Kulgemeyer and Wittwer (2022) provide a definition of the same that differentiates between two criteria of how an instructional explanation is perceived by the learners and two criteria of how learners perceive their own understanding of the topic:

An illusion of understanding is the mistaken belief that (1) an instructional explanation is (a) a scientifically correct and (b) a high-quality explanation in terms of comprehensibility, and (2) the learners themselves (c) understood the concept, and (d) do not require further instruction.

That is close to Paik and Schraw’s (2013) view of the illusion of understanding hypothesis; they also include characteristics of (1) the material and of (2) the learners in their concept. Research provides several possible factors that influence these four criteria with respect to explainer videos. For example, Lowe (2003) suggests that graphical animations, in general, support an illusion of understanding. Graphical animations may turn a learner’s focus on attention-grabbing information rather than relevant principles that are explained. In an early work in this field, Salomon (1984) found that learners using video feel that the medium is more efficient than print, while participants learning in print outperformed the video group in an inference-making test. Wiley (2019) calls this a potential ‘seduction effect’ of images. Kulgemeyer and Wittwer (2022) found that presenting misconceptions, close to everyday experiences, as alternative explanations is highly convincing and fosters the illusion of understanding. They also highlight that misconceptions constitute a major educational obstacle in those who learn science from explainer videos.

A difference between the perceived learning gain and the objective learning gain may be more prevalent in learning from digital media when compared to learning from print media. Studies show that students who read print outperform those reading digital media in reading comprehension tests, while the digital media group believes they score higher (Dahan Golan et al., 2018; Singer Trakhman et al., 2019). Reading print also seems more efficient than listening to the same information in a podcast; however, learners prefer podcasts (Daniel & Woody, 2010).

Researchers have identified a particular tendency for learners from print to outperform those learning from video (Walma van der Molen & van der Voort, 2000). For complex principles (such as those that physics learning would require), different studies show an advantage in the case of print (DeFleur et al., 1992; Furnham et al., 1987; Gunter et al., 1986; Walma van der Molen & van der Voort, 2000). In a more recent work, Lee and List (2019) found an advantage to learning from text, while List and Ballenger (2019) did not find a clear difference. Similarly, Zinn et al. (2021) compared text and video in technical education and did not find a difference between the media. Merkt et al. (2011), however, provide evidence that the reason may lie in the level of interactivity; videos with interactive elements are equally as effective as print. This further underlines the need to integrate explainer videos into the cognitive activities described above.

However, the cognitive theory of multimedia learning suggests that learning via two channels (e.g. by learning from graphical representations that are commented on by a narrator) can, under certain conditions, help to overcome the limitations of working memory (Mayer, 2001). A key criterion for success is that learners be cognitively activated and use their prior knowledge to integrate the information from both channels. Explainer videos tend to fulfil these criteria and, being a multimodal form of information presentation, may be superior to textual explanations.

In a nutshell, results regarding learning from video compared to print media are ambiguous, with a tendency towards finding text to have an advantage. However, it is not clear which medium is better, including explanations in physics.

Recent research has shown that the context of learning from instructional explanations is also important for, both, actual learning and the potential illusion of understanding. Salmerón et al. (2020) deem the so-called ‘shallowing hypothesis’ important when learning with explanations from online videos. It refers to the fact that such interactions in everyday contexts are often superficial and oriented toward quick rewards such as ‘likes’ (Annisette & Lafreniere, 2017). Indeed, processing the information presented in a digital medium may follow this superficial approach (Salmerón et al., 2020). Also, the shallowing hypothesis may have the advantage of written explanations when compared to explainer videos with respect to achievement.

The basic idea is that the medium overshadows how the explanation is dealt with. Thus, a medium learners are used to just superficially interact with might lead to the mistaken impression that an explanation was easy and they understood it while that was actually not the case. Explainer videos, therefore, might lead to an illusion of understanding.

However, the underlying idea of the shallowing hypothesis might also suggest a potential illusion of understanding when learning from written explanations, especially when they appear like formal textbooks. Contrary to their interaction with explainer videos, learners may tend to not superficially interact with explanations. With respect to such explanations, they interact first and foremost in a formal teaching context, which may lead to written explanations from textbooks appearing to be a more reliable source. Therefore, they may believe to have learned more from textbook entries than from explainer videos about the same subject because the information seems more trustworthy and this impression overshadows the actual achievement.

When it comes to comparing written explanations and explainer videos, there is no clear evidence that one can be used over the other in science classrooms. In particular, online explainer videos have a broad range of qualities because anyone can upload content on online platforms like YouTube (Bråten et al., 2018; Kulgemeyer & Wittwer, 2022). However, while several studies in educational psychology suggest that written explanations are at least as efficient as explainer videos, it is unclear whether explainer videos foster an unjustified belief of understanding – or, in other words, if science learners using explainer videos feel more competent than learners using written explanations from textbooks. As described above, a potential illusion of understanding may highly influence how learners perceive their lessons classroom lessons and, in particular, how open they are to further instruction. This forms the premise of the present study.

Materials and methods

Research questions

Learners with low prior knowledge are often considered the main target group for instructional explanations in general (Acuna et al., 2011; Wittwer & Renkl, 2008). Based on the aforementioned theoretical considerations, we identify a desideratum in the research on the effects of physics explainer videos versus those of written explanations that appear textbook-like on learners with low prior knowledge.

First, based on the ambiguous evidence from prior studies, it is unclear whether an explainer video or a written explanation has an advantage in terms of achievement.

Regarding a possible illusion of understanding, previous research suggests that video and, in particular, the animations and images contained in the videos have a ‘seductive effect’ (Wiley, 2019) which makes learners overestimate their knowledge and regard further instruction as unnecessary. However, the ‘shallowing hypothesis’ reminds us that the medium matters in one’s perception of an explanation. Written explanations that appear to be textbook entries give the impression of being more reliable and convincing, which also results in learners overestimating their knowledge gains.

Therefore, the research questions of the present study are:

1. Do explainer videos foster declarative knowledge when compared to textbook-like written explanations?

2. Do explainer videos foster an illusion of understanding when compared to textbook-like written explanations?

Since, as described above, the evidence from prior studies is ambiguous, it is not possible to predict which medium has an advantage in terms of achievement. Thus, in an attempt to answer Question 1, we test the following hypotheses:

Hypothesis 1: The medium of an explainer video leads to greater achievement of declarative knowledge than written explanations. Alternative hypothesis 1a: Textbook-like written explanations lead to greater achievement of declarative knowledge than explainer videos. Alternative hypothesis 1b: Explainer videos and written explanations lead to equal achievement of declarative knowledge.

As described above, an illusion of understanding is the false belief that a topic has been understood and no further instruction is required, but also that the explanation is scientifically correct and well-constructed. Similarly to Research Question 1, there is ambiguous evidence regarding Research Question 2. Thus, the following hypotheses regarding Research Question 2 will be tested:

Hypothesis 2: The medium of an explainer video leads to a greater illusion of understanding than written explanations. Alternative hypothesis 2a: Textbook-like written explanations lead to a greater illusion of understanding than explainer videos. Alternative hypothesis 2b: Explainer videos and written explanations lead to an equal illusion of understanding.

We choose to use the same video as Kulgemeyer and Wittwer (2022) on the introduction of the concept of force as its duration is rather short (2:23 min), and also because a reliable related test instrument exists.

Kulgemeyer (2018a) suggests that the learning progress from explainer videos was primarily limited to declarative knowledge directly presented in the explanation. For conceptual knowledge, which includes flexibly transferring the newly learned principles to various other examples not directly included in the explanation, more active learning followed by learning tasks is likely to be required. Therefore, our study addresses declarative knowledge as a construct of interest.

Context

The study took place in the introductory physics course for pre-service elementary teachers at a German university. The course is mandatory for first-semester students and covers basic school physics. The experiment was conducted before they were introduced to the concept of force. It was mandatory to participate; however, pre-service teachers could opt not to have their data used for further analysis.

Participants and design

The sample consisted of 181 learners with 31 opting out to have their data used. 150 learners remained in the sample. As it is often the case, none of them had opted to enrol in a physics course in the last three years of school, the academic preparation phase of the German school system. Thus, they can be expected to have low prior knowledge in physics, comparable to that of secondary school students. The sample, therefore, is a convenience sample, however, a suitable one for an exploration because their characteristics that make them different from learners at school being introduced into the concept of force are not expected to highly impact the dependent variables of the study (Gollwitzer et al., 2017).

Of these 150 learners, 18 identified themselves as male, 132 as female, and none as non-binary. This is not unusual since primary education is a subject that is, at least in Germany, predominantly elected by female pre-service teachers. Their final school exam grade was, on average, M = 2.37 (SD = 0.490) located on a scale from 1 (‘very good’) to 6 (‘unsatisfactory’). In the German state of Northrhine-Westfalia, where the course was taught, the average final school exam grade was 2.43 in 2020¹ which does not differ significantly from the sample (t(105) = −1,34; p = .18). A random generator assorted the learner to two groups: (1) video (n_V = 78) and (2) written explanation (n_W= 72).

Materials: explainer video and written explanation

For this study, we decided to use the explainer video from Kulgemeyer and Wittwer (2022), which introduced the concept of force. It has been designed following a framework for explainer videos that has been empirically researched for validity (Kulgemeyer, 2018a, 2018c). This framework has emerged from a systematic literature review regarding the criteria for successful instructional explanations with a focus on physics education (Geelan, 2012; Kulgemeyer, 2018a; Wittwer & Renkl, 2008). It is in line with other frameworks on the quality of explainer videos, for example, Brame (2016) and Findeisen et al. (2019). The framework includes the seven factors influencing the effectiveness of explainer videos (Kulgemeyer, 2018a): (1) the structure of the video, (2) adaptation to a group of addressees, (3) the use of appropriate tools to achieve adaptation, (4) minimal explanations that avoid digressions and keep the cognitive load low, (5) highlighting that the explanation itself is relevant to the learner and the other highly relevant parts, (6) providing learning tasks following the video, and (7) a focus on a scientific principle. These seven factors can be manipulated using 14 different features (Table 1).

Table 1. Framework for effective explanation videos (from Kulgemeyer (2018a)).

Factors	Feature	Description
Structure	1.Rule-example, Example-rule	Rule-example structure if the learning goal is factual knowledge. Example-rule structure if the learning goal is a routine or procedural knowledge.
	2.Summarising	The video summarises the explanation.
Adaptation	3.Adaptation to prior knowledge, misconceptions, and interest	The video is adapted to a group of addresses and their potential knowledge, misconceptions, or interests. To do so, it uses the ‘tools for adaptation.’
Tools for adaptation	4.Examples	The video uses examples to illustrate a principle.
	5.Analogies and models	The video uses analogies and models that connect the new information with a familiar area.
	6.Representation forms, demonstrations	The video uses representation forms and/or demonstrations.
	7.Level of language	The video uses a familiar level of language.
	8.Level of mathematization	The video uses a familiar level of mathematization.
Minimal Explanation	9.Avoiding digressions	The video focuses on the core idea, avoids digressions and keeps the cognitive load low.
	10.High coherence	The video connects sentences with connectors, especially ‘because.’
Highlighting relevancy	11.Highlighting relevancy	The video explicitly highlights why the explained topic is relevant to the explainee.
	12.Direct addressing	The explainee is addressed directly, e.g. by using the second-person singular.
Learning tasks	13. Learning tasks	The video describes learning tasks the explainees can engage into actively.
Principles	14.New, complex principle	The video focuses on a new science principle that is too complex to understand by self-explanation.

Kulgemeyer and Wittwer (2022) report based on an expert rating that these criteria have been fulfilled in the explainer video. The video is designed to be comparable to online explainer videos (e.g. as those found on YouTube). Therefore, this study addresses learning from such of videos in a laboratory setting.

To test the effect of the medium, the content should remain unchanged. The written explanation, therefore, includes just the script of the video, i.e. the literal transcript (Figure 1). This written explanation is presented combined with two images directly from the video, as the presence of images is a prerequisite for a good explanation (Table 1). All the features included in Table 1 are, therefore, included in the written explanation to a comparable degree as in the video. The written explanation, including the illustrations, covers 1.5 pages of text and is presented in a digital format, not in print. The readability coefficient LIX works well with the German language (Lenhard & Lenhard, 2014) and indicates low text complexity (LIX = 35.6).

Figure 1. Overview of the materials used (in German as originally used). Left: screenshot of the video, right: illustrated written explanation using the same text and illustrations as used in the video (1 of 1,5 pages) (video and text by Erber, 2019).

Table 2 provides a brief overview of the script of the video. The content of the textbook page is identical (translated from the German language). Both the video (2:23 min) and the textbook (1.5 pages) were comparable under the criteria of Kulgemeyer (2018a). They followed a rule-example structure with a summary at the end. They made use of examples (e.g. a rolling golf ball) and representation forms (e.g. pictures with stick figures). The levels of mathematisation and language were identical. They focused on the concept (minimal explanation) and avoided digressions. They used prompts to highlight the most important parts and addressed the explainee directly.

Table 2. Parts of the script of the explainer video and the textbook page, translated from German language. Representation forms are not included, the used examples are just summarised briefly. Adapted from Kulgemeyer and Wittwer (2022).

Video script / textbook

[Pointing out that force is an important concept in physics, giving overview of the video.] In a nutshell, force is the reason why an object accelerates. A force always acts with a certain ‘strength’ in a particular direction. This means that an object will not accelerate if no force acts on it. Of course, in that case, the object may rest, but it may also move with a constant velocity. In both cases, the object would not accelerate and, therefore, in both cases no force acts on the object. How strong the acceleration is, given that a certain force is acting on an object depends on the mass of the object. If it has a lower mass, the acceleration will be higher. If it has a higher mass, the acceleration will be lower. This idea on force can be used to describe all kinds of moving objects. [Example: Applying this idea on the example of a golf ball that stops moving after it gets hit because of the force of friction.] Let us sum up: A force causes an acceleration of an object in a certain direction. It does not matter if the object rests or moves—if not force acts on it, it does not accelerate. [Ending the video with a learning task: use the explained ideas to explain why a bicycle accelerates if you go downhill.]

Measures

We used the same instruments as those used by Kulgemeyer and Wittwer (2022). Next to demographics, we measured (1) control variables (gender, final school grade, experience with explainer videos, academic self-concept physics [all pre-test]), (2) the declarative knowledge that was part of the explanation (pre- and post-test), and (3) the illusion of understanding (just post-test).

Control variables

A core aspect of the control variables was the final school exam grade (‘Abitur’). This grade is a good indicator of one’s success in academics (e.g. Binder et al., 2021) and, therefore, reflects an academic skill set that is useful for learning from both textual and video content. This should be comparable between the groups. We also measured the gender of the participants and the academic self-concept in physics (5-point Likert scale, six items, α = 0.82, e.g. ‘I get good grades in physics’; scale taken from Kulgemeyer & Wittwer, 2022).

The experience with explainer videos was measured using 12 different items to obtain insights into various possible reasons for using explainer videos and the learner’s perception of explainer videos in general. All the items were 5-point Likert scales ranging from 1 (total agreement) to 5 (total disagreement). The items included the experience (e.g. ‘I have often watched explainer videos regarding school or university topics’), the perception of videos on YouTube (e.g. ‘I usually select the explainer video with the most number of likes’), and possible reasons for watching explainer videos (e.g. ‘I watch explainer videos to prepare for exams’). The primary purpose of these items was to ensure comparability between the experimental conditions. The items were not expected to form a coherent scale; thus, no reliability was reported.

We decided against including similar items to measure the experience with written explanations and textbook entries. Other than with explainer videos, we did not expect any student to leave school without any experience with textbooks, and the likelihood that both groups vary in that characteristic was very low compared to the danger of a longer test instrument harming the test motivation (also because they were assigned randomly to an experimental condition). This decision has been discussed as a limitation of the study in the discussion section.

Declarative knowledge on the concept of force

The scale for declarative knowledge of the concept of force was originally designed by Kulgemeyer and Wittwer (2022) by analyzing the script of the explainer video. To ensure content validity, explicit learning opportunities need to exist in both the explainer video and the written explanation for all items in the test instrument. The final scale consisted of 11 multiple-choice items. A sample item is shown in Figure 2. The reliability of the scale is sufficient, both, in the pre-test (α = 0.77) and the post-test (α = 0.78). The same scale has been used in Kulgemeyer and Wittwer (2022) and is motivated by instruments such as the force concept inventory (Hestenes et al., 1992). Since the test items mirror the learning opportunities in video and text, the test instrument was designed for learners with low prior knowledge (Figure 2). More arguments concerning validity can be found in Kulgemeyer and Wittwer (2022).

Figure 2. Sample item for conceptual knowledge and misconceptual knowledge (translated from the German language, taken from Kulgemeyer and Wittwer (2022)).

Illusion of understanding

To measure the illusion of understanding, we used the instrument of Kulgemeyer and Wittwer (2022) that covers all four criteria for an illusion of understanding mentioned above: An illusion of understanding is the mistaken belief that (1) an instructional material is (a) a scientifically correct and (b) a high-quality explanation in terms of comprehensibility, and (2) the learners themselves (c) understood the concept, and (d) did not require further instruction.

That is close to the understanding of an illusion of understanding as described by Paik and Schraw (2013). The final scale consisted of 7 items. The reliability was found to be sufficient (α = 0.76). Six items were 5-point Likert scale items ranging from 1 ‘totally agree’ to 5 ‘totally disagree’. Sample items would be ‘I need to learn more on the concept force’, ‘I would advise a friend to prepare with this video/text for an exam’, or ‘I understood what force means in physics’. In one item we asked them to estimate how many of the test items they thought they could solve before each, pre-test and post-test. The post-test item was included into the scale.

In the present article, we use the following wording: the scale is called the ‘illusion of understanding scale’ while the score on this scale is called a more neutral ‘belief of understanding’. Following the research reported above, the belief of understanding can be qualified as an illusion of understanding when it does not reflect objectively measured understanding. The score, therefore, is called an ‘illusion of understanding’ when it has been empirically proven that there is a difference between the belief and actual understanding. How this has been carried out in the analysis is discussed in the following section.

Overview of the study, procedure, and analysis

The study followed an experimental design (Figure 3). Testing time was limited to 45 min.

Figure 3. Overview of the study.

After a short introduction, the learners were assigned to either the explainer video group or the written explanation group by a random generator. Afterwards, they were given a pre-test including demographics, experience with explainer videos, and declarative knowledge on the concept of force. Then, they worked with the medium according to the experimental conditions they had been categorised into. They took the test individually and were not allowed to browse the internet or use any material other than the medium they had been given.

The pre-test and post-test instruments were identical between the groups, except for the phrasing of some statements in the perceived explanation quality scale matching their respective medium (e.g. ‘the video is well explained’ versus ‘the text is well-explained’). Both groups were tested online in a testing environment based on LimeSurvey. The video and written explanations were embedded in this online testing environment.

We analyzed the learning gains of the two groups (hypothesis 1 and alternatives). We used t-tests, one-way ANOVAs and ANCOVAs to compare the results from the pre-tests and the post-tests of declarative knowledge, and the scores on the Illusion of Understanding scale. Subsequently, we compared the correlations between declarative knowledge and the scores from the Illusion of Understanding for each experimental condition in order to gain insights into the question of whether the learning gains and the belief of understanding were associated.

In case hypothesis 2 was true, we would expect a significant difference between the experimental conditions regarding their illusion of understanding scores, while the actual learning progress was comparable. In addition, we expected their belief in understanding to show no correlation to the actual declarative knowledge since it was an illusion and not a realistic view of their understanding.

Results

Descriptive statistics

The descriptive statistics of the study variables, including the control variables, are presented in Table 3.

Table 3. Descriptive Statistics of Study Variables.

	Video Group	Text Group
	M (SD)	M(SD)	Range
Declarative Knowledge (Pre)	0.55 (0.19)	0.55 (0.17)	0–1
Declarative Knowledge (Post)	0.70 (0.15)	0.67 (0.16)	0–1
Illusion of Understanding	0.59 (0.07)	0.63 (0.09)	0–1
Control Variables
Final school grade	2.37 (0.47)	2.37 (0.51)	1–6
Academic self-concept physics (pre)	0.27 (0.12)	0.30 (0.11)	0–1
Academic self-concept physics (post)	0.29 (0.11)	0.30 (0.11)	0–1
‘I have often watched explainer videos regarding school or university topics’	2.14 (1.14)	1.89 (0.99)	1–5
‘I usually find an explainer video that helps if I am just searching for it’	2.17 (0.87)	1.97 (0.65)	1–5
‘I usually select the explainer video that gets the most number of likes’	3.37 (1.07)	3.56 (1.03)	1–5
‘The video with the most number of likes usually explains the best’	3.36 (0.97)	3.24 (0.88)	1–5
‘Usually, explainer videos on YouTube are scientifically correct’	3.04 (0.73)	2.94 (0.84)	1–5
‘I watch explainer videos to prepare for exams’	0.91 (0.29)	0.92 (0.28)	0–1
‘I watch explainer videos for the purpose of entertainment’	0.06 (0.25)	0.03 (0.17)	0–1
‘I watch explainer videos to solve learning tasks’	0.91 (0.29)	0.93 (0.26)	0–1
‘I watch explainer videos to prepare for my own teaching as a teacher’	0.06 (0.25)	0.10 (0.30)	0–1
‘I watch explainer videos to prepare for seminars and lectures’	0.35 (0.48)	0.39 (0.49)	0–1
‘I usually do not watch explainer videos at all’	0.09 (0.29)	0.04 (0.20)	0–1

Note. M = mean; SD = standard deviation. Range statements on reasons to watch the explainer video: 0 = no, 1 = yes

Control variables

Using t-tests, we did not find any significant differences between the groups regarding the control variables (Table 1). Most importantly, gender (χ²(1) = 0.10, p = 0.75), the final school grade (t(104) = 0.01, p = 0.99, d = 0.00), and the academic self-concept in physics (pre: t(148) = 1.15, p = 0.25, d = 0.19; post: t(142) = 0.56, p = 0.57, d = 0.09) did not differ. That is a prerequisite to compare the groups.

Research question 1: do explainer videos foster declarative knowledge when compared to textbook-like written explanations?

We did not find a difference between the groups in their performance in the post-test (F(1,148) = 1.55, p = 0.215, η²= 0.010) or the pre-test (F(1,147) = 0.048, p = 0.828, η²= 0.000) on declarative knowledge. Figure 4 shows a comparison of this scale. The learning gain (ANCOVA of the post-test results adjusted for the pre-test scores) did not reveal any differences either (F(1,147) = 1.70, p = 0.20, partial η²= 0.011). These results are in line with alternative hypothesis 1b.

Figure 4. Comparison of the illusion of understanding and declarative knowledge in the post-test between the groups. Reported are effect sizes from ANCOVAs (illusion of understanding: adjusted for declarative knowledge (post); declarative knowledge: adjusted for pre-test).

Research question 2: do explainer videos foster an illusion of understanding when compared to textbook-like written explanations?

Hypothesis 1: greater illusion of understanding from explainer videos

The written explanation group scored higher than the video group on the illusion of understanding scale with a small effect (F(1,148) = 5.87, p = 0.017, η²= 0.038; Figure 4). Also, an ANCOVA adjusted for the knowledge in the post-test (to control the objective declarative knowledge into the analysis) reveals this difference (F(1,147) = 6.65, p = 0.011, partial η²= 0.043).

To gain further insights into how strongly the belief of understanding and the actual understanding were associated, we conducted a correlational analysis. A positive correlation between the actual understanding and belief of understanding would hint to a realistic picture of one’s own performance. We did not find a correlation between the belief of understanding and declarative knowledge in the post-test (video group: r = 0.08, p = 0.49; written explanation group: r = −0.16, p = 0.17). Following Gollwitzer et al. (2017, p. 547), this difference between the correlation coefficients is not significant (Z = 1.45, p = 0.148). Overall, these results support the alternative hypothesis 2a.

Discussion

We did not find a difference between the explainer video group and the written explanation group in their performance with respect to their achievement or performance on the post-test on declarative knowledge. This result is in line with alternative hypothesis 1b and aligned with prior research on media comparison regarding print and video, which has yielded ambiguous results. In our study, video was explicitly not the superior medium in terms of achievement.

The results, overall, are in line the alternative hypothesis 2a. The written explanation group was significantly more convinced that they understood the topic and did not require further instruction. The video group was more critical of their own performance. To label this as an illusion of understanding, we need evidence to identify this result as an erroneous belief. Two arguments support the conclusion of an illusion of understanding:

1. The first argument comes from a theoretical perspective and deals with learning the concept of force. From what we know about this, it may be argued that both groups are most likely to require further instruction to fully understand the complex concept of force. It is well known in physics education that this particular concept is difficult to grasp because of multiple misconceptions and because a conceptual change is very hard to accomplish in this context (Alonzo & Elby, 2019). A short video or a short text is most probably not sufficient. Also, as science educators, we would like our students to be sufficiently critical toward themselves so that they remain open to further instruction. The written explanation group was significantly more convinced of their understanding. Since (as argued), it is likely that they require further instruction, this is likely a false belief and, therefore, an illusion of understanding.

2. The second argument comes from an empirical perspective. First of all, the ANCOVA of the illusion of understanding scale adjusted for knowledge in the post-test reveals a significantly higher belief of understanding in the written explanation group. In addition, both groups score between 67% and 70% (with no ceiling effect) in a declarative knowledge test that was designed only for learners with low content knowledge (Kulgemeyer & Wittwer, 2022). Both groups, however, are significantly positive about their understanding. That suggests an illusion of understanding. Also, the correlational analysis revealed no relationship between test performance in the post-test and the belief of understanding for both groups. This supports the assumption that none of the groups has a realistic belief of understanding.

In a nutshell, we want to reiterate that neither group reflected well on their actual learning gains, but the written explanation group gave a worse performance in this respect.

Based on these two arguments, we posit that both groups have an illusion of understanding, but the written explanation group has a higher degree of an illusion of understanding. The medium of written explanation (appearing as a textbook entry) is the most probable, but not the only, reason for this (cf. ‘study limitations’).

Overall, our study shows that written explanations similar to textbook entries can potentially lead to an illusion of understanding. As discussed above, the reason might come from an inverse ‘shallowing hypothesis’ (Salmerón et al., 2020). The ‘shallowing hypothesis’ suggests that learners transfer their strategies of superficial everyday interaction with online content to learning with digital media. However, this may also explain why they felt they learnt more from the written explanation. As per the normative tendency, they may not only superficially interact with textbook-like explanations as they barely have strategies to interact with written explanations; with such explanations, they interact first and foremost within the context of formal teaching. Thus, students may perceive textbook entries as more convincing because they appear more formal, closer to past classroom experiences, and closer to what their teachers advise them to learn from. This may affect how they perceive the content of an explanation in this format, resulting in an illusion of understanding. Novice physics learners, in particular, may be distracted by the superficial appearance as they lack the skill to critically evaluate the content. Written explanations might appear to be more reliable than explainer videos, even though the actual achievement is not necessarily superior (as in the present study). However, the effect is rather small. Both groups show signs of an illusion of understanding.

Study limitations

There are several limitations to the present study. First, the results were derived in the context of physics and, in particular, for only one topic in the subject. We cannot safely assume that for other domains or topics, the results will be the same. In addition, a limitation of the design is that learning skills – including reading and video comprehension skills – were not assessed in this study. This was partly due to restrictions on the testing time, and also because learning from text and video involves a multitude of academic skills, and not all of them could have been using a single test instrument. Both video and text were rather short which might underestimate the effects of the media, maybe longer videos outperform longer texts. On the other hand, shorter material probably reduced the impact of reading or video comprehension skills on the results.

We decided that our approach is suitable for an explorative study. We used the final school exam grade as a measure for such academic skills useful to learning from both text and video and, additionally, to measure their experience with explainer videos. Both did not differ between the groups. The participants had average final school exam grades. That indicates average academic skills including reading skills, especially because the average final school grade (the ‘Abitur’) is a well-known predictor of one’s success in academics (Binder et al., 2021). However, it still is a convenience sample that does not fully mirror average learners at schools. Overall, the results should be treated as first insights and should not be generalised. Also, we conducted an experimental laboratory study, and external validity, therefore, was rather low.

The sample size (N = 150) might have resulted in the study being insufficiently powered, limiting the significance of the correlational analysis between the actual understanding and belief of understanding. We conducted a post hoc power analysis using G*Power (Erdfelder et al., 1996) that revealed that, for the size observed in the present study (r = −0.16, |ρ| = −0.3), the power to detect an effect of this size was 0.97 (with α = 0.05), which is above the recommended power of 0.8 (Cohen, 1988). Thus, even a very small effect for the written explanation group is unlikely.

Furthermore, the null result regarding the achievement of declarative knowledge may be due to a lack of statistical power. Therefore, we additionally conducted a post hoc power analysis. The effect size of this ANOVA of the learning gains was η²= 0.01 (small effect following Cohen (1988)). The power to detect an effect of this size was determined to be 0.23 with α set at 0.05. Thus, we cannot completely rule out a small effect. However, this effect would have been far below the desired effect size for education (e.g.. Hattie, 2009). For medium effect sizes (η²= 0.06), the power was determined to be 0.86, above the recommended power of 0.8. We therefore regard our study as having adequate statistical power. The sample size is already relatively large for an experiment in science education research and very small effects might not be meaningful. Therefore, the power analyses do not affect our interpretations of the outcomes.

Study Implications

First of all, our results support the growing evidence that learning with digital media is not automatically superior in terms of achievement.

We see no reason that written explanations from textbooks should not remain a part of physics teaching. Also, we argue that our results highlight the potential dangers of leaving students to their own devices with either explainer videos or written explanations. Both groups show signs of an illusion of understanding. Prior research also indicated that learning with explanations, in general, works only when the explanations are well-embedded in ongoing cognitive activities such as learning tasks (Wittwer & Renkl, 2008). Perhaps working with learning tasks helps one assess their own performance better. Employing such learning tasks that build on the content of an instructional explanation may, therefore, serve as a countermeasure to the effect of written explanations on the illusion of understanding. Further research is needed to gain more insights into learning through explanations using different media. However, we will reiterate that an illusion of understanding has potentially harmful consequences for physics learning (Acuna et al., 2011; Kulgemeyer & Wittwer, 2022): if learners are convinced they have fully understood a topic, they may reject further teaching, deeming it redundant and irrelevant. This way, they are less cognitively activated (which is a core dimension of instructional quality in science). Also, if they get to choose further instructional material (e.g. in self-directed learning) they might stop after they are convinced of their understanding and that might apply to both explainer videos and written explanations. This underscores the fact that physics teaching does not stop with providing an explanation in a particular medium; further learning tasks are required to make learners aware of what they do not understand. Further studies exploring the effects of an illusion of understanding are required.

Ethical statement

The study met the ethics/human subject requirements of the University of Paderborn at the time the data was collected.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

1 https://de.statista.com/statistik/daten/studie/36277/umfrage/durchschnittliche-abiturnoten-im-vergleich-der-bundeslaender/