Temporal synchrony between speech, action and gesture during language production

Abstract

Researchers have theorised that speech and gesture are integrated in communication. We ask whether this integrated relationship is indexed by a unique temporal link between speech and gesture. University students performed videotaped tasks that elicited: (1) speech and action descriptions about how to act on objects and (2) speech and gesture descriptions about how to act on objects. Integration was indexed by measuring the onset of speech with actions and speech with gestures (in milliseconds), with smaller differences reflecting a greater degree of synchrony. One hundred per cent of the subjects gestured in the gesture condition and performed actions in the action condition. Speech and gesture were more tightly synchronised than speech with action. The greater synchrony between gesture and speech suggests that the two could be uniquely designed to work together for the purpose of communication.

Keywords:

• gesture
• action
• speech
• language
• synchrony

Acknowledgements

We would like to acknowledge all those college students who participated in this study, the Psychology departments at Northeastern Illinois and Colgate Universities and the Center for Language and Brain at Colgate University for their generous support. We would also like to thank David McNeill who was the inspiration for this study and provided continuous support with the interpretation of the data and many other individuals who have helped shape our thinking about gesture, in particular, Susan Goldin-Meadow, Martha Alibali, Sotaro Kita and Melissa Singer. We would like to particularly thank Theodora Koumoutsakis for her invaluable help with the preparation of this manuscript.

Notes

¹ The descriptions often included statements about the sequence of steps leading to the primary action (e.g., when asked to describe how to throw a dart, a subject said, ‘first I would pick up the object and then I would place my right foot forward and then throw the dart’. In addition, some subjects felt compelled to comment on the result saying something like, ‘…in order to hit the target’. We focused on the primary target action, gesture and verb related to the request given by the researcher. In the example just described, we coded the onset of the verb ‘throw’ and the onset of the accompanying action or gesture for throwing (see Appendices B and C for examples of these descriptions).