With task experience students learn to ignore the content, not just the location of irrelevant information

ABSTRACT

Presentation of irrelevant additional information hampers learning. However, using a word-learning task, recent research demonstrated that an initial negative effect of mismatching pictures on learning no longer occurred once learners gained task experience. It is unclear, however, whether learners consciously suppressed attention to the content of the mismatching pictures. Therefore, we examined the effects of a picture location change towards the end of the learning phase: for half of the participants, the picture location was changed after they gained task experience. If participants only ignore the location of mismatching pictures, word learning in the mismatched condition should be hampered after the location change. Changing the location of the mismatching pictures did not affect recall in the mismatched condition, but, surprisingly, the location change did hamper learning in the matched condition. In sum, it seems that participants learned to ignore the content, and not just the location of the irrelevant information.

KEYWORDS:

• Multimedia learning
• attention
• word learning
• task experience

The “multimedia effect” indicates that learning improves when study tasks or materials combine pictorial and verbal representations of the content (Butcher, 2014). However, this beneficial effect on learning only occurs when both representations are crucial for understanding the subject at hand. When one source of information is extraneous, that is, not relevant for learning, it will hinder learning (Kalyuga & Sweller, 2014; Mayer & Fiorella, 2014). For example, learning is hampered when interesting information is added to enrich materials (i.e. seductive details, e.g. Harp & Mayer, 1998); when learning materials are unnecessarily elaborate, presenting textual explanations with self-explanatory diagrams (e.g. Chandler & Sweller, 1991), or providing details and examples whereas a concise summary would suffice (e.g. Mayer, Bove, Bryman, Mars, & Tapangco, 1996); or when information on related systems is presented when learning about a specific system (Mayer, DeLeeuw, & Ayres, 2007).

The negative effects of extraneous information on learning arise because learners attend to, process, and attempt to integrate the extraneous information with the essential information, which unnecessarily depletes working memory resources required for learning (Mayer, 2014; Sweller, Ayres, & Kalyuga, 2011). Moreover, in some cases, the content of the additionally presented information may actively interfere with learning the essential information (e.g. Mayer et al., 2007). However, eye-tracking studies have shown that participants learn to ignore extraneous information with task experience (Haider & Frensch, 1999) or explicit instruction (Hegarty, Canham, & Fabrikant, 2010). Therefore, task experience might be a boundary condition to the negative effect of extraneous information on learning: If people learn to ignore such information with task experience, it should no longer hamper learning.

A recent study yielded evidence in line with this hypothesis (Rop, Van Wermeskerken, De Nooijer, Verkoeijen, & Van Gog, 2016). Participants learned the definitions of 15 words (from an artificial language called Vimmi; see Macedonia & Knösche, 2011) in 3 blocks of 5 words, with a recall test after each block. After the first block, recall performance was lower when words were coupled with mismatching pictures than with matching pictures; however, once participants had some experience with the task (i.e. in Blocks 2 and 3), the mismatching pictures no longer hampered recall performance compared to the matching pictures (Experiment 2). A follow-up experiment, employing eye-tracking methodology to study learners’ attention allocation, showed that learners adapted their study strategy with increasing task experience and started to ignore the mismatching pictures more strongly than the matching pictures.

Because the mismatching pictures always appeared at a fixed location, it is an open question whether learners consciously suppressed attention to the pictures because they were aware that the content was irrelevant for the task at hand. One way to answer this question is by systematically changing the location of the pictures for half of the participants after they have accumulated task experience (i.e. in the third block of words; see Figure 1 for an impression of the location change). If they learned to suppress attention to the location, learning should be negatively affected in the mismatched condition with a location change (because the change reinstates attention to the pictures, at least briefly) compared to all other conditions. However, if participants learned that the content is irrelevant, they would be expected to actively suppress attention to the pictures regardless of the location and performance should not be lower in the mismatched condition with a location change compared to all other conditions.

Figure 1. Example materials. The spoken definition (e.g. ifra means to polish or scrape with sandpaper) is presented twice, the second time accompanied by a picture. In the matched (1a) and mismatched (1c) conditions, this picture was always presented underneath the word. In the matched-change (1b) and mismatched-change (1d) conditions the picture was presented underneath the word in Blocks 1 and 2, but the location of the picture changed in Block 3, in which the picture was now presented above the word. [To view this figure in color, please see the online version of this journal.]

Another possibility is that a location change will only briefly hamper learning. This hypothesis is based on the signal-suppression hypothesis (Gaspelin, Leonard, & Luck, 2015; Sawaki & Luck, 2010), which states that a combination of bottom-up and top-down influences determines attention paid to a stimulus. While a location change might briefly attract attention due to saliency of a stimuli unexpectedly appearing at a different location (bottom-up attention influence, cf. Remington, Johnston, & Yantis, 1992), awareness that the stimulus does not contain useful content (top-down attention influence) would suppress attention to the picture. Consequently, in our learning task, the location change of the mismatching pictures might only hamper learning for the first few words.

Present experiment

In the present experiment, participants learned fifteen word definitions in three blocks of five words, with either matching (depicting the action to be learned) or mismatching (depicting another action) pictures added. In two conditions, the pictures were presented underneath the word during the whole experiment (these conditions replicate the conditions in Rop et al., 2016), while in the other two conditions the location of the pictures changed in Block 3, in which they were now presented above the word.

We hypothesised that if learners are aware that the mismatching pictures are irrelevant for their learning, they would suppress attention to the pictures even after the location changes, in which case the change would not influence word learning (either in Block 3 as a whole or for the first few words) compared to all other conditions. If they only ignored the location, however, recall performance in the mismatched condition should be negatively affected after the location change (at least for the first few words in Block 3). We also performed a direct replication experiment (Experiment 1b) as one finding from Experiment 1a was interesting but surprising.

Method

Participants and design

Participants (Experiment 1a: n = 429, Experiment 1b: n = 485) were recruited on Amazon’s Mechanical Turk (Buhrmester, Kwang, & Gosling, 2011) and were paid 1.50 US dollar for their participation. A-priori defined post hoc exclusion criteria were: Being left handed (n = 67, n = 80); being a non-native English speaker (n = 11, n = 4); participating in a noisy environment (i.e. a self-reported score of 7 or higher on a 9-point scale, n = 5, n = 5); and taking notes during the learning phase (n = 8, n = 10). Furthermore, some participants were excluded for misunderstanding the instructions (i.e. they wrote down the names of the pictures instead of the word definitions which they were instructed to learn; n = 8, n = 8); and some participants were excluded because they encountered technical difficulties (n = 2, n = 4). Finally, one participant in Experiment 1a did not have an MTurk ID and was excluded, while in Experiment 1b we excluded all participants that already participated in Experiment 1a (n = 22).

This left 327 participants in Experiment 1a (M_age = 37.50 years, SD = 11.71 years, range 18–68; 199 females), who were randomly distributed over 4 conditions resulting from a 2 × 2 design with between-subjects factors “Picture Match” (matching vs. mismatching) and “Location Change” (yes vs. no): matching pictures no location change (matched condition, n = 72), matching pictures with location change (matched-change condition, n = 90), mismatching pictures no location change (mismatched condition, n = 87), and mismatching pictures with location change (mismatched-change condition, n = 78). In Experiment 1b, 352 participants were left (M_age = 36.25 years, SD = 11.02 years, range 18–71; 180 females), who were randomly distributed over the matched (n = 91), matched-change (n = 89), mismatched (n = 86), and mismatched-change (n = 86) conditions.

Materials and procedure

The learning materials were programmed in Qualtrics software (Qualtrics, Provo, UT). Participants learned the definitions of fifteen Vimmi words in three blocks of five words, with a recall test after each block. Each word was coupled to the definition of an action verb (e.g. “ifra” means “to polish or scrape with sandpaper”). Participants saw the word printed on screen and heard the spoken definition of the word they had to learn twice (each presentation lasted 11 seconds and the program automatically progressed). A matching or a mismatching picture accompanied the word the second time participants heard the definition. In the two conditions without a location change, the picture was always presented underneath the word. In the two change conditions, the picture was presented underneath the word in Blocks 1 and 2, but above the word in Block 3 (see Figure 1).

Participants’ knowledge of the definition was tested with a cued recall retention test after each block, in which they were presented with the written word and had to type in the associated definition as literally as possible.¹ A block always consisted of the same 5 words, but the order of the blocks was randomised using a Latin-square design, which resulted in 12 lists used for the experiment. There were no breaks between blocks. The experiment lasted about 20 minutes.

Scoring

Participants were awarded 1 point if they provided a complete definition on the cued recall test (e.g. “to polish or scrape with sandpaper” for the word “ifra”). When part of the definition was missing, they received 0.5 point (e.g. “to polish”). If they did not provide a definition, or if it was completely wrong, 0 points were awarded (e.g. “to remove something written by wiping” which was the definition of another word in that block). So, every participant could score a maximum of five points on each test. A random subset of the data (11.0% in Experiment 1a and 10.2% in Experiment 1b) was scored by a second rater, and interrater reliability was high (κ = .91 in Experiment 1a and κ = .84 in Experiment 1b).

Results

In all analyses, a significance level of .05 was maintained, and when the sphericity assumption was violated, the Greenhouse–Geisser correction is reported. Effect size measures used were partial eta-squared and Cohen’s d. Both can be interpreted in terms of small (η_p² ∼ .01, d ∼ 0.2), medium (η_p² ∼ .06, d ∼ 0.5), and large (η_p² ∼ .14, d ∼ 0.8) effect sizes (Cohen, 1988). First, to check whether we could replicate prior findings by Rop et al. (2016) that performance is initially hampered by mismatching pictures, we performed a mixed ANOVA on recall performance with Word Block (first or second) as within-subjects factor and Picture Match (matching or mismatching) as between-subjects factor. Then, to test our hypothesis concerning the effects of the location change on recall performance we conducted 2 × 2 ANOVAs with Picture Match (matching or mismatching) and Location Change (yes or no) as between-subjects factors on recall performance in Block 3. Finally, we investigated effects on word level within Block 3 by calculating the recall performance per word in that block and performing two repeated-measures ANOVA’s (for the mismatched and matched condition separately), with Serial Position (1, 2, 3, 4, and 5) as within-subjects factor and Location Change (yes or no) as between-subjects factor.

Check: did mismatching pictures initially hamper learning?

Table 1 shows the results on recall performance in Blocks 1 and 2 of Experiments 1a and 1b, and of Experiment 2 of Rop et al. (2016). Both Experiments 1a and 1b showed a significant main effect of Word Block indicating that recall performance improved in both conditions from Block 1 to Block 2 (1a: F(1, 325) = 10.76, p = .001, η_p² = .03; 1b: F(1, 350) = 15.58, p < .001, η_p² = .04). However, we did not replicate the interaction between Word Block and Picture Match that was found in the study by Rop et al. (2016; 1a: F(1, 325) = 1.17, p = .347, η_p² < .01; 1b: F(1, 350) = 1.86, p = .174, η_p² = .01). Because the pattern in the data seemed consistent with our hypothesis and the interaction effect was small in the prior study, we decided to analyse the combined data from the prior and current study in order to get an estimate of the combined effect of these three studies. To do so, we performed a mixed ANOVA with Picture Match (matching or mismatching) and Experiment (Rop et al., Experiment 2; Experiment 1a, and Experiment 1b from the present study) as between-subjects factors and Word Block (first or second) as a within-subjects factor. In this analysis, the interaction between Word Block and Picture Match was significant, F(1, 777) = 6.11, p = .014, η_p² = .01, while the three-way interaction Word block × Picture Match × Experiment was not, F < 1. The lack of a three-way interaction suggests that the patterns of results in the experiments are comparable. Therefore we followed-up on the interaction between Word Block and Picture Match with one-tailed t-tests. These tests showed that participants in the matched condition had better recall performance than participants in the mismatched condition in Block 1, t(781) = 2.31, p = .011, d = 0.17, but not in Block 2, t(781) = 0.06, p = .475, d < 0.01.

Table 1. Mean (and SD) recall performance (max. = 5) as a function of picture match and location change in Experiment 1a, Experiment 1b, and Rop et al., Experiment 2.

	Block 1	Block 2	Block 3
Experiment 1a (n = 327)
Matched	3.04 (1.50)	3.17 (1.65)	3.24 (1.63)
Matched-change	2.89 (1.38)	3.17 (1.53)	2.85 (1.54)
Mean	2.96 (1.43)	3.17 (1.58)
Mismatched	2.71 (1.57)	3.09 (1.53)	3.16 (1.52)
Mismatched-change	3.01 (1.49)	3.40 (1.54)	3.49 (1.46)
Mean	2.86 (1.54)	3.23 (1.54)
Experiment 1b (n = 352)
Matched	2.58 (1.39)	2.77 (1.48)	2.73 (1.52)
Matched-change	2.63 (1.85)	2.90 (1.85)	2.52 (1.54)
Mean	2.61 (1.63)	2.83 (1.67)
Mismatched	2.42 (1.41)	2.65 (1.43)	2.73 (1.49)
Mismatched-change	2.23 (1.39)	2.94 (1.45)	3.01 (1.42)
Mean	2.32 (1.40)	2.79 (1.44)
Rop et al., Exp 2 (n = 104)
Matched	3.15 (1.46)	3.21 (1.51)	3.14 (1.63)
Mismatched	2.51 (1.51)	3.17 (1.45)	3.13 (1.44)

Hypothesis: is recall performance in Block 3 affected by the picture location change?

Experiment 1a

The recall performance in Block 3 is shown in Table 1. There was no main effect of Picture Match, F(1, 323) = 2.66, p = .104, η_p² = .01, or Location Change, F < 1 on recall performance, but we did find a significant interaction between Picture Match and Location Change, F(1, 323) = 4.61, p = .032, η_p² = .01. Bonferroni corrected follow-up t-tests (two-tailed) indicated that, in the absence of a change, recall performance between the matched and mismatched conditions was comparable, t(157) = 0.35, p > .999, d = 0.06, 95% CI for the difference in means = [−0.58; 0.41]. Surprisingly, after a change, recall performance was higher in the mismatched condition than in the matched condition, t(166) = 2.77, p = .012, d = 0.43, 95% CI = [0.18; 1.10].

Experiment 1b

Again, there was no main effect of Picture Match, F(1, 348) = 2.31, p = .129, η_p² = .01, or Location Change, F < 1, and—in contrast to Experiment 1a—the interaction effect was not statistically significant, F(1, 348) = 2.28, p = .132, η_p² = .01, although the pattern of results as well as the effect size was comparable to Experiment 1a. Therefore, we exploratively conducted the same set of Bonferroni corrected follow-up tests as in Experiment 1a. These results were also in the same direction as in Experiment 1a, although not statistically significant. In the absence of a change, recall performance between the two picture conditions was comparable, t(175) = 0.01, p > .999, d < 0.01, 95% CI = [−0.45; 0.45], while recall performance seemed higher in the mismatched condition than in the matched condition after a location change, t(173) = 2.16, p = .066, d = 0.33, 95% CI = [0.04; 0.93].

Combined analysis

We ran a combined analysis of Experiments 1a and 1b², as these experiments are a direct replication of each other. We performed a 2 × 2 × 2 ANOVA with Picture Match (matching or mismatching), Location Change (yes or no) and Experiment (Experiments 1a and 1b) as between-subjects factors. This analysis revealed a significant interaction between Picture Match and Location Change, F(1, 671) = 15.52, p = .009, η_p² = .01, while the three-way interaction Picture Match × Location Change × Experiment was not significant, F < 1 (again suggesting that the Experiments are comparable). The follow-up tests showed that, in the absence of a change, recall performance between the two picture conditions did not differ, t(334) = 0.07, p = .994, d = 0.01, 95% CI = [−0.34; 0.32], while recall performance was higher in the mismatched condition than in the matched condition after a location change, t(341) = 3.39, p = .001, d = 0.37, 95% CI = [0.23; 0.87]. This combined analysis gives a better estimation of the true effect of a location change, which is a small-to-medium effect.

Hypothesis: is recall performance in Block 3 affected on word level?

Experiment 1a

Table 2 presents the recall performance data at the word level in Block 3. Our main objective of this analysis was to explore whether a negative effect of location change would occur in the first few serial positions of Block 3 for mismatching pictures but not for matching pictures. Therefore, we will only report on the interaction between Location Change and Serial Position, which was not significant for the matched, F < 1 and mismatched condition, F < 1.

Table 2. Mean (and SD) recall performance on the words in Block 3 as a function of picture match and location change in Experiments 1a and 1b.

	Experiment 1a				Experiment 1b
	Matched		Mismatched		Matched		Mismatched
	No change	Change	No change	Change	No change	Change	No change	Change
1	.70 (.43)	.66 (.46)	.66 (.46)	.69 (.43)	.63 (.45)	.60 (.45)	.56 (.47)	.71 (.43)
2	.60 (.46)	.52 (.43)	.59 (.47)	.69 (.44)	.49 (.42)	.43 (.42)	.51 (.45)	.55 (.43)
3	.60 (.45)	.51 (.42)	.57 (.43)	.60 (.42)	.47 (.42)	.44 (.42)	.48 (.44)	.48 (.43)
4	.64 (.42)	.53 (.45)	.60 (.43)	.64 (.43)	.48 (.42)	.47 (.41)	.48 (.46)	.56 (.44)
5	.71 (.39)	.63 (.44)	.74 (.38)	.88 (.30)	.67 (.41)	.60 (.46)	.70 (.40)	.71 (.39)
∑	3.24 (1.63)	2.85 (1.54)	3.16 (1.52)	3.49 (1.46)	2.73 (1.52)	2.52 (1.54)	2.73 (1.49)	3.01 (1.42)

Experiment 1b

Again, we did not find an interaction between Location Change and Serial Position for both conditions: matched, F < 1; mismatched, F(3.63, 616.67) = 1.21, p = .307, η_p² = .01.

Explorative analysis: does recall performance within conditions change from Blocks 2 to 3?

To exploratively follow up on the unexpected finding that recall performance was higher in the mismatched that in the matched condition when a change was present (see Table 1), we performed Bonferroni corrected paired t-tests to compare the performance in Blocks 2 and 3 in all four conditions of Experiment 1a and 1b. The results of Experiment 1a suggest that recall performance was lower in Block 3 compared to Block 2 in the matched-change condition, t(89) = 2.11, p = .072, d = 0.21, 95% CI = [−0.02; −0.61], whereas performance remained stable across Blocks 2 and 3 in the other three conditions, minimum p = .860, maximum d = 0.07. In Experiment 1b, again, there seemed to be a performance drop in the matched-change condition from Blocks 2–3, t(88) = 2.07, p = .084, d = 0.22, 95% CI = [−0.01; −0.74], which did not occur in the other conditions, minimum p < .999, maximum d = 0.05.

Discussion

Prior research has shown that presenting learners with extraneous information that is irrelevant for the task at hand, hampers their learning (Kalyuga & Sweller, 2014; Mayer & Fiorella, 2014). However, a recent study comparing the effect of matching and mismatching pictures on word learning, suggested that task experience might be a boundary condition to this effect (Rop et al., 2016). The negative effect on learning was present initially but no longer occurred once learners gained task experience, because they started to ignore the irrelevant information. However, the mismatching pictures always appeared at a fixed location. Therefore, it was unclear whether learners consciously suppressed attention to the pictures because they were aware that the content was irrelevant for the task at hand. The aim of the present study was to address this question by systematically changing the location of the pictures for half of the participants after they have accumulated task experience. Our results indicated that changing the picture location influenced recall performance, albeit in an unexpected way. The location change in Block 3 resulted in poorer recall in the matched condition compared to the mismatched condition, and an explorative follow-up analysis suggested that recall performance decreased in the matched condition from Block 2 to Block 3, while it remained stable in all other conditions. Note that this analysis was not statistically significant after Bonferroni correction. However, the effect sizes in Experiment 1a and Experiment 1b were almost equal (d = 0.21 and 0.22). Combined, these findings suggest that changing the location of matching pictures seemed to have a small negative effect on word learning.

Eye-tracking data from the study by Rop et al. (2016) showed that matching pictures continuously attracted a substantial amount of attention, from an average of 76% of fixation time in Block 1 to 60% in Block 3, over the course of the experiment. Thus, in the present study, when the location of these pictures suddenly changed in Block 3, participants might have wondered why the location of the pictures changed, which would distract from learning the definitions. This distraction might have hampered learning as participants focused more on the changing picture location, and less on encoding the actual definition. Future research could address the plausibility of this explanation by measuring learners’ visual attention allocation using eye-tracking methodology to see whether they anticipated on the picture appearing in the other location, or by interviewing them after the experiment. More importantly for our hypotheses, the fact that the location change did not affect performance in the mismatched condition suggests that students were aware that the content was irrelevant for their learning of the word definitions and that they continued to ignore these pictures.

Limitations and future research

A limitation of the present study is that we did not directly measure visual attention allocation, but the performance data suggest that the mismatching pictures must have been consciously ignored via top-down influences, because otherwise a drop in performance compared to Block 2 would have occurred. Furthermore, within Block 3 we did not find a negative effect of mismatching pictures on the first words, so even if the location change attracted learners’ attention initially (i.e. stimulus-driven, bottom-up influences; cf. Remington et al., 1992), it seems to have been suppressed quickly (cf. Gaspelin et al., 2015; Sawaki & Luck, 2010). Possibly, participants were able to ignore the mismatching pictures by redirecting their attention to the artificial language word that was shown on the screen (in the study by Rop et al., 2016, attention to the word increased from an average of 42% in Block 1 to 69% in Block 3 in the mismatched condition).

Another possible limitation of the present study could be that we only replicated the initial finding that mismatching pictures have a negative effect on learning compared to matching pictures when we combined the results of multiple experiments. Note though, that the pattern of means of recall performance was consistent over all experiments: Participants learning with mismatching pictures score lower in Block 1 than participants learning with matching pictures. Secondly, because we were able to include multiple experiments in the combined analysis, we had a large sample size, which means that we can be fairly certain that the effect exists, although it is small. Finally, the small effect size is consistent with prior studies using the same materials (De Nooijer, Van Gog, Paas, & Zwaan, 2013; Rop et al., 2016) and may perhaps be due to the relatively low complexity of the learning materials. All things considered, we can regard this replication attempt a modest success, although the effect size for the crucial interaction we found is much smaller than the effect size reported in Rop et al. (2016). Future research should address whether these findings would replicate with more complex multimedia learning materials, such as expository texts combined with explanatory pictures. Such materials might induce larger effect sizes, and would provide evidence that task experience is a robust boundary condition to the negative effects of irrelevant information on learning.

Practical implications

Our results may also be relevant for educational practice. Although the study by Rop et al. (2016) already showed that over time, students are able to adapt their study strategy and ignore irrelevant information, it was an open question whether participants consciously suppressed attention to the pictures. The results of the present study suggest that they truly learned to ignore the content, and not just the location of the irrelevant information. Because information that is relevant for novices might become irrelevant for advanced learners, it is important for instructional designers to know that students seem to be able to adapt their study strategies in multimedia learning. Interestingly, our findings do suggest that instructional designers might want to be careful with changing the location of relevant information after learners have gained experience with the task, as our findings suggest that this can have a (small) negative effect on learning. Future research should attempt to replicate these findings in other materials, however, before clear instructional design guidelines can be derived.

Disclosure statement

No potential conflict of interest was reported by the authors.

ORCID

Gertjan Rop http://orcid.org/0000-0001-6204-1607

Additional information

Funding

This research was funded by a Research Excellence Initiative grant from the Erasmus University Rotterdam awarded to the Educational Psychology section.

Notes

1 We also explored whether there were differences in experienced cognitive load among the conditions, by asking participants to indicate how much mental effort they invested in learning the words on a nine point rating scale (Paas, 1992), ranging from one (very, very low effort) to nine (very, very high effort). Because of word limits we do not report these data (the only significant finding concerned a main effect of Picture Match in Experiment 1b, F(1, 348) = 5.96, p = .015, ηp2 = .02, indicating that participants in the mismatched condition invested more mental effort).

2 Based on an anonymous Reviewer’s suggestion.