In which direction to move? Facilitative and interference effects of gestures on problem solver's thinking

ABSTRACT

Recent research shows that co-speech gestures can influence gesturers’ thought. This line of research suggests that the influence of gestures is so strong, that it can wash out and reverse an effect of learning. We argue that these findings need a more robust and ecologically valid test, which we provide in this article. Our results support the claim that gestures not only reflect information in our mental representations, but can also influence gesturer's thought by adding action information to one's mental representation during problem solving (Tower of Hanoi). We show, however, that the effect of gestures on subsequent performance is not as strong as previously suggested. As opposed to what previous research indicates, gestures' facilitative effect through learning was not nullified by the potentially interfering effect on subsequent problem-solving performance of incompatible gestures. To conclude, using gestures during problem solving seems to provide more benefits than costs for task performance.

KEYWORDS:

• Gestures
• problem solving
• embodied cognition
• Tower of Hanoi
• mental representation

People often gesture when they communicate. Co-speech hand gestures reflect what is said, but also reveal information not explicitly conveyed in spoken language (Goldin-Meadow, 2003). This shows that gestures are an indication of what goes on in the speaker's mind. Interestingly, converging evidence shows that just as how thought influences gestures, gestures also affect speaker's cognitive processes (Hostetter & Boncoddo, 2017; Kita, Alibali, & Chu, 2017; Pouw, de Nooijer, van Gog, Zwaan, & Paas, 2014). For example, it has been shown that co-speech hand gestures support speech production (Kita, 2000; Krauss, Chen, & Gottesman, 2000), spatial problem solving (Alibali, Spencer, Knox, & Kita, 2011), and learning to solve mathematical equations (Broaders, Cook, Mitchell, & Goldin-Meadow, 2007).

In recent experiments, Goldin-Meadow and colleagues identified another cognitive effect, suggesting that co-speech hand gestures influence subsequent performance on a manual problem-solving task by strengthening sensorimotor routines for manipulating task-relevant objects (Beilock & Goldin-Meadow, 2010; Cooperrider, Wakefield, & Goldin-meadow, 2015; Goldin-Meadow & Beilock, 2010; Trofatter, Kontra, Beilock, & Goldin-Meadow, 2015). With this remarkable finding, they were the first to show that, like actions, gestures can influence thought. The general paradigm in these studies involves participants solving the Tower of Hanoi (TOH; Newell & Simon, 1972). Its goal is to move a stack of discs from one of three pegs to another, by moving one disc at a time. The discs are arranged by size (i.e. largest on bottom; smallest on top) and larger discs cannot be placed on smaller discs. Beilock and Goldin-Meadow (2010) used this task to investigate gestures' effect on cognition. Participants (Experiment 1) solved the TOH (TOH1), after which they explained the solution with gestures. Then, participants solved the TOH again (TOH2), which was either exactly the same as TOH1 (No-Switch Condition) or with switched disc weights (Switch Condition); rather than having weights corresponding with their sizes, the smallest disc was heaviest whereas the largest disc was lightest. Unlike TOH1, the smallest disc could thus no longer be lifted with one hand. Results indicated that the more one-handed grasping gestures participants used during their explanation, the slower they solved TOH2 in the Switch condition. Participants in the Switch condition, but not in the No-Switch condition, performed worse on TOH2 than on TOH1. Additional experiments showed that a no-explanation group (Experiment 2; Beilock & Goldin-Meadow, 2010) or an action group (i.e. performing TOH1 again instead of explaining; Goldin-Meadow & Beilock, 2010) generally solved TOH2 faster than TOH1 in the Switch and No-Switch conditions. Two recent studies, ruling out factors such as visual feedback from gestures (Cooperrider et al., 2015) and influences of accompanying speech (Trofatter et al., 2015) in explaining this effect, replicated these general results. Importantly, these findings extend prior research by demonstrating that specific gestures adding specific action information to a specific mental representation subsequently have a specific influence on performance.

These findings suggest that describing physical interactions with the environment using gestures strengthens relevant information involving action and environment—including object affordances like weight—as part of gesturers' mental representation. Motivated by the embodied-cognition framework—claiming that information not present in the environment can be internally represented by mentally simulating real-world actions (Wilson, 2002)—Beilock and Goldin-Meadow (2010) explain the above findings by arguing that “gesture not only is a vehicle for expressing action information, but also, because it is itself an action, can add action information to the gesturer's mental representations.” (p. 1605). If this added information is incompatible with later actions, even when irrelevant to task performance (here: weight), performance is hindered. Strikingly, gestures incompatible with the subsequent task (e.g. one-handed gesture for the smallest, heaviest disc) apparently interfere with performance so strongly that they can nullify positive effects of gesture on learning (Beilock & Goldin-Meadow, 2010). That is, the interfering effect of incompatible gestures is stronger than the facilitating effect generally induced by practice (i.e. learning effect), resulting in slower performance. Notably, as gestures show a larger effect on cognition than action, it appears that stronger mental representations are constructed when the relevant objects are absent as opposed to present (Trofatter et al., 2015).

Although the above studies (Beilock & Goldin-Meadow, 2010; Cooperrider et al., 2015; Goldin-Meadow & Beilock, 2010; Trofatter et al., 2015) provide converging findings following from an elegant paradigm, the methodology also invites for a more ecologically valid and robust test. One aspect cautioning definitive conclusions on the magnitude and generalizability of the reported effects is the weight manipulation. Firstly, encountering a TOH2 task in the Switch condition visually similar to TOH1, that turns out to have disc weights mismatching their respective sizes when acted upon, causes a feeling of surprise. It is likely that this influences participants' solution time (i.e. dependent variable). Possibly, this surprise effect exacerbates the slowdown in performance, explaining the provocative finding that participants in the Switch condition solve TOH2 even slower than TOH1. Additionally, the weight-induced hand moves (one-handed versus two-handed grasps) may be confounded with solution time because the number of one-handed and two-handed movements differs between TOH1 and TOH2. Since the smallest disc is typically moved more often than the largest disc, more two-handed movements are required in solving TOH2 in the Switch condition than in the No-Switch condition and during TOH1. Two-handed movements are generally performed with lower velocity than one-handed movements (Asai, Sugimori, & Tanno, 2010), resulting in longer solution times. Also, two-handed movements increase task complexity as one has to rely solely on imagination when thinking ahead for the next move instead of having one hand available to support imagining the next move (Hinckley, Pausch, Proffitt, & Kassell, 1998), which may further exacerbate this confound. Note, however, that such a confound does not explain the reported difference between gesture versus no-gesture conditions. Finally, in the described experiments, disc weight was a task-irrelevant object property. If gestures indeed influence subsequent performance by strengthening sensorimotor information, even larger effects should be expected when manipulating task-relevant properties (e.g. movement direction). This also increases the ecological validity of the manipulation (i.e. it is unlikely that a TOH in practice has discs with reversed weights), providing more generalisable data.

A second aspect cautioning a definitive conclusion regarding the presence of a cognitive effect of gesture is the robustness of the reported results. Firstly, the findings of the prior studies together suggest that gesturing adds information to the speaker's mental representation, rather than merely reflecting the information that it already contains. Therefore, participants who do not gesture do not show a decrement in performance when the disc weights are switched. None of the previous studies, however, has directly compared (in a single analysis) a gesture condition to a no-gesture control condition. Secondly, prior results are based on small participant samples. In all four papers published on this topic, a maximum sample size of 14 participants per condition was used. For enough power, however, to detect actual effects, each condition should at least contain 20 observations (Simmons, Nelson, & Simonsohn, 2011). In a between-subject design this is even more important due to the potential effects of selection bias. To determine the magnitude and reliability of the reported effects, a more robust test of gesture's ability to strengthen a mental representation of sensorimotor information is necessary.

Present experiment

To address the above concerns, in the present experiment we used the paradigm and procedure from the Beilock and Goldin-Meadow (2010) study and adapted it by manipulating solving direction of the task, rather than disc weight. In our Switch condition, participants were required to solve TOH2 from right-to-left, instead of from left-to-right (i.e. direction in TOH1). Not only is this a more ecologically valid manipulation than switching disc weights, it is also unlikely that a surprise effect impacts solution time, as participants can observe the reversed solving direction in the Switch condition before they start solving TOH2. Performance, however, may be hindered by gestures that have enriched the mental representation with action information in the opposite direction (e.g. gestures representing movement of discs from left to right). Manipulating solving direction also ensures that the required movements are similar between TOH1 and TOH2, assuming that left-to-right movements do not differ in execution time from right-to-left movements (Kelso, Southard, & Goodman, 1979).

Additionally, to ensure sufficient power (a problem in the previous studies using this paradigm), this study employed more than 20 participants per condition. In contrast to previous studies, the present design allows for a direct comparison between all conditions. The effects of explaining with gestures on subsequent performance (Switch and No-Switch conditions) are directly compared to a no-explanation control condition. A no-explanation condition rather than an explaining-without-gestures condition was chosen, as it has been shown that people routinely gesture when talking about solving the TOH task (Garber & Goldin-Meadow, 2002). Furthermore, in accordance with the pilot study conducted by Beilock and Goldin-Meadow (2010), we have observed that participants told not to gesture generally experience difficulties fully explaining their moves. Rather than explaining, participants in the no-explanation condition were asked to read a short article (see also Experiment 2 in Beilock & Goldin-Meadow, 2010).

We hypothesise that, based on the above-mentioned studies, performance on the subsequent task (TOH2) will be hindered when participants' gestures contain action information (left-to-right movements) incompatible with the movements made at TOH2 in the Switch condition (right-to-left movements) as compared to performance on TOH2 in the No-Switch condition. If this interference is strong enough to nullify an effect of learning, performance on TOH2 (only in the Switch condition) should be slower than performance on TOH1. If not, only a smaller learning effect is expected in the Switch condition than in the No-Switch condition.

Method

Participants

In total, 101 undergraduate students at the Vrije Universiteit Amsterdam were randomly assigned to one of four conditions. Prior to analyses, solution time scores for TOH1 and TOH2 larger than 3 times the interquartile range per condition were considered extreme outliers and were removed from the dataset (n = 7). The age of the 94 remaining participants with complete data (45 males) ranged from 18 to 30 years (M_age = 22.43 years, SD = 2.80). Groups (no switch/no explanation: n = 25; no switch/explanation with gestures: n = 21; switch/no explanation: n = 25; switch/explanation with gestures: n = 23) did not differ regarding the proportion of gender (χ² < 1) and mean age (F < 1). All participants reported to have had no prior exposure to the TOH task. Participants were rewarded with course credit.

Design and procedure

All participants provided informed consent before they were tested individually in a quiet room. In accordance with the procedure used by Beilock and Goldin-Meadow (2010), there were four practice TOH trials in which the first three used the three-disk version. The fourth practice trial used the four-disk version, which was also used for the two following experimental trials. After each practice trial, the TOH was placed out of sight and participants were asked to explain how they solved the TOH task. They were encouraged to use their hands during explaining.

After the four practice trials, participants were asked to solve the four-disc version of the TOH (TOH1). The time to solve the task and the number of movements required to complete the task were recorded and served as the baseline to which the second experimental trial was compared. No participants were excluded based on the solution time for TOH1.¹ Following prior studies using the same paradigm (e.g. Beilock & Goldin-Meadow, 2010), after completing the first experimental trial (i.e. TOH1), the TOH was taken away so that it was out of participants' sight. Half of the participants (explanation-with-gestures condition) were then asked to explain how they solved the task to the test leader. They were encouraged to use their hands. The other half (no-explanation condition) was asked to read a short article. To ensure that the reading task was taken seriously, participants were told that they would be tested after the next TOH trial (i.e. TOH2). The time between TOH1 and TOH2 was approximately the same for both the explanation-with-gesture condition and the no-explanation condition. For participants in the explanation-with-gesture condition, left-to-right movement gestures were calculated as a percentage of the total number of gestures representing direction (i.e. left to right and right to left).

Finally, all participants were asked to solve the four-disc TOH task (TOH2) again. For participants in the no-switch condition, this trial was exactly the same as the previous trials (starting position: all disks on the left peg; moving direction: left-to-right). In the switch condition, the TOH was turned around (starting position: all disks on the right peg) so that participants would solve the task in the opposite direction (moving direction: right-to-left). Following TOH2, participants in the no-gesture condition were told they did not have to answer any questions regarding the article. All participants were debriefed.

Results

Solution time in seconds was highly correlated to the number of movements (r = .87 at TOH1 and r = .86 at TOH2). Therefore, and in accordance with the procedure adopted by Beilock and Goldin-Meadow (2010), a univariate analysis was conducted with solution time as the dependent variable. A three-way mixed analysis of variance on solution time was conducted with Task (TOH1 vs. TOH2) as within-subject factor, and Switch (switch vs. no switch) and Explanation (explanation with gesture vs. no explanation) as between-subject factors. All ANOVA results are shown in Table 1. As can be seen from Figure 1, there was a significant main effect of Task. This indicates that overall there was a learning effect from TOH1 to TOH2 as participants had faster solution times for TOH2 (M = 52.99, SD = 22.00) than for TOH1 (M = 66.58, SD = 29.43). This learning effect (TOH2-TOH1) appeared larger for participants in the Explanation-with-gesture group (M_difference = −19.84, SD = 28.29; p < .001) than for participants who did not explain the task (M_difference = −8.09, SD = 25.28; p = .028), which was evidenced by the significant interaction between Task and Explanation. Interestingly, no significant interactions were found for the Switch factor (Fs < 1). This shows that in the switch condition, where the direction of the TOH was changed from left-to-right to right-to-left, the learning effect was not cancelled out by explaining the task with gestures that were opposite to the direction needed to perform TOH2. However, when testing the positive correlation between the percentage of gestures from left-to-right and the change in solution time (TOH2-TOH1) in the switch- and no-switch groups, similar findings were obtained as those reported by Beilock and Goldin-Meadow (2010). The more incompatible gestures (i.e. left-to-right) the switch group produced, the more they interfered with their performance and thus, reduced the learning effect (r = .40, p = .030, Figure 2, right panel). This relation was not found for the no-switch group² (r = .27, p = .121, Figure 2, left panel).

Figure 1. Solution times in seconds from TOH1 (gray bars) to TOH2 (white bars) as a function of experimental group. The left graph shows the results for participants who did not explain between solving TOH1 and TOH2 and the right graph shows the results for participants that explained how they solved the task using gestures before solving TOH2. For both groups, results are shown separately for participants in the switch and no-switch condition.

Figure 2. Change in solution time (TOH2 – TOH1) as a function of the percentage of left-to-right movement gestures, used to explain how TOH1 was solved. Results are shown separately for participants in the switch (right) and no-switch (left) condition.

Table 1. Analyses of variance for solution times on TOH task in seconds (df = 90).

Analysis	Effect	F	p		90%  CI
Overall	Task	24.91**	<.001	.22	.102;  .330
	Task × Switch	0.00	.981	.00
	Task × Explanation	4.39*	.039	.05	.001;  .133
	Task × Switch × Explanation	0.09	.770	.00

*p < .05, **p < .001.

Discussion

Our study provides a more robust test of previous provoking findings suggesting that co-speech hand gestures influence subsequent performance on a manual problem-solving task by strengthening sensorimotor routines (Beilock & Goldin-Meadow, 2010; Cooperrider et al., 2015; Goldin-Meadow & Beilock, 2010; Trofatter et al., 2015). To exclude potential confounders of solution time, the experimental manipulation did not focus on disc weight, which was exclusively done in prior research, but on solving direction (left-to-right versus right-to-left). The present study further extends previous studies by having a larger sample size and providing a direct comparison between an explanation-with-gesture and no-explanation group.

Overall, results showed a reliable learning effect, meaning that participants generally solved TOH2 faster than TOH1. More importantly, this learning effect was stronger for the explanation-with-gesture group than for the no-explanation group. This suggests that explaining with gesture has a facilitating effect on performance, functioning as an effective form of practice. This is consistent with research showing that gesture about action is even more powerful than action itself (Goldin-Meadow & Beilock, 2010), supposedly because stronger mental representations are constructed when the relevant objects are absent as opposed to present (Trofatter et al., 2015).

The most striking findings relate to performance differences between participants in the switch and no-switch conditions. In the work by Goldin-Meadow and colleagues, moving the smallest, lightest disc in TOH1 could be represented with one-handed gestures, whereas the smallest but heaviest disc in the switch-condition at TOH2 could not be lifted with one hand. They found that incompatible gestures interfered so strongly with acting upon the switched TOH2 that participants solved this task even slower than TOH1 (Beilock & Goldin-Meadow, 2010; Cooperrider et al., 2015; Goldin-Meadow & Beilock, 2010; Trofatter et al., 2015). So, the facilitating effect of practicing by explaining with gesture (note that task demands were exactly the same for TOH1 and TOH2) was completely nullified. Interestingly, we could not replicate these findings with our data. Rather, the magnitude of the learning effect did not differ for the switch and no-switch groups, indicating that explaining the TOH1 from left-to-right with compatible gestures did not interfere with subsequent incompatible movements (i.e. solving TOH2 from right-to-left). This suggests that earlier reported interfering effects of gestures regarding this task were likely exacerbated due to methodological issues.

Alternatively, not replicating the previous findings could be due to differences in the gestures that were measured. Although weight is mainly represented by character viewpoint gestures—gestures resembling real-world actions as they incorporate the speaker's body—such as one-hand vs. two-hand holding of discs, direction is mainly represented by observer viewpoint gestures, which do not simulate the character's actions, but result from simulated object properties such as the trajectory of an object (McNeill, 1992). It could be that character viewpoint gestures have a stronger influence on one's mental representation than observer viewpoint gestures, explaining the lack of an effect in the present study (for a similar discussion, see Goldin-Meadow & Beilock, 2010).³ Future research is warranted to empirically test this explanation, which could be done by directly comparing the differential effects that these two types of gesture have on people's mental representation. As participants in our control condition neither gestured nor explained the TOH task, another suggestion for future research is to apply a research design that differentiates between the effects of gesture and verbal explanation.

When looking more closely at the percentage of left-to-right gestures, our data replicated earlier findings—extending it to another task dimension, i.e. solving direction—showing that the more incompatible gestures were used, the larger the interference with subsequent performance (Beilock & Goldin-Meadow, 2010; Trofatter et al., 2015). This interference resulted in a relatively smaller learning effect. When participants explained how they solved TOH1, they usually did not talk about specific directions using words like “left” or “right” as indicated by informal observations during the experiment.⁴ Rather, they used their hands to indicate direction, accompanied by ambiguous speech like “this disc goes there” or “move this to the end”. Therefore, solving direction was mostly reflected in gesture, but not in speech, and particular gestures impacted TOH2 solution time.

Together, our results support the claim that gestures not only reflect information in our mental representations of a task, but can also influence gesturer's thought by adding action information to one's mental representation. If this added information is incompatible with subsequent actions, it hinders performance. Although our results are largely in line with prior findings and explanations, they indicate that the effects of gestures may not be as strong as previously suggested. Hindrance from incompatible gestures on subsequent performance seems to remain weaker than the opposed facilitating effect of practicing a task by explaining with gesture. To conclude, explaining a task while using gestures seems a good way of practice, even more so than redoing the task. It is important, however, that gestures are compatible with subsequent task performance to ensure the benefit of gestures.

Acknowledgments

We thank Rosalie Kaandorp, Stella Knol, Smahane Bouzeya, and Duygu Dulkadir for their help in data collection.

Disclosure statement

No potential conflict of interest was reported by the authors.

ORCID

Stephanie I. Wassenburg http://orcid.org/0000-0002-9007-3136

Notes

1 Beilock and Goldin-Meadow (2010) excluded participants who completed the TOH1 task faster than 65 s because they argued that these participants had little room to improve and, therefore, it would be impossible to test for learning. In our sample, however, 61.4% of the participants completed TOH1 under 65 s, with a total sample mean of 67.19 s. In addition, almost all participants (including the fast participants) completed TOH2 faster than TOH1. Because for the present study we were interested in the effect of gestures on all participants’ performance, and to take the experiment's power into account, we decided to only report analyses on the total sample of participants. Importantly, we did run the analyses excluding participants with TOH1 scores <65 s, and these analyses yielded the same pattern of results.

2 One participant in the no-switch group was excluded from the correlational analysis, because their percentage of left-to-right movement gestures (42%) appeared to be an extreme outlier (more than 3 times the interquartile range).

3 The authors thank an anonymous reviewer for suggesting this alternative explanation.

4 Note that we also coded the available video data (25%) for directional language. More than half of the participants in these videos never mentioned direction. Participants who did indicate direction in language, mentioned both “left” and “right” equally often, and directional language did not correlate significantly with solving time.