When actions speak too much louder than words: Hand gestures disrupt word learning when phonetic demands are high

Abstract

It is now widely accepted that hand gestures help people understand and learn language. Here, we provide an exception to this general rule—when phonetic demands are high, gesture actually hurts. Native English-speaking adults were instructed on the meaning of novel Japanese word pairs that were for non-native speakers phonetically hard (/ite/ vs. /itte/, which differ by only a geminate) or easy (/tate/ vs. /butta/, which differ by a geminate and also their segmental composition). The words were presented either with or without congruent iconic gestures, for example, “Ite means stay” (with a STAY gesture). After instruction, participants were given phonetic and vocabulary tests for the words they had learned. Although performance for the phonetic task was above chance for all conditions, gesture played different roles in the semantic task for easy and hard word pairs—it helped word learning for easy pairs, but it hurt for hard pairs. These results suggest that gesture and speech are semantically integrated during word learning, but only when phonetic demands are not too high.

Keywords:

• Hand
• Gesture
• Iconic
• Beat
• Speech
• Language
• Semantic
• Word
• Phoneme
• Phonetic
• Prosodic
• Geminate
• Foreign
• L2
• Learning
• Memory
• Vocabulary

Acknowledgements

The authors would like to thank Yukari Hirata for consulting with us on early phases of this project, reading drafts of the manuscript and helping us with the revision. We also thank Satomi Nunokawa for serving as the speaker (gesturer) in our instructional stimuli. We also thank the members of the Nijmegen Gesture Center at the Max Planck Institute for Psycholinguistics (Nijmegen, The Netherlands) for feedback on the data.

Notes

¹For simplicity, the paper will refer to “hard” and “easy” word pairs. However, it should be noted that these word pairs are easy or hard only for non-native speakers of Japanese who do not use geminates to make a phonological distinction; whereas for native Japanese speakers—who makes these distinctions on a regular basis—these two types of word pairs are equally easy.

²The means are presented in actual proportions even though arcsine transformed proportions were used for the ANOVAs.

³Because the analyses by participants and items produced inconsistent results, we do not treat this main effect as reliable.