Development of multimodal discourse comprehension: cohesive use of space by gestures

Abstract

This study examined how well 5-, 6-, 10-year-olds and adults integrated information from spoken discourse with cohesive use of space in gesture, in comprehension. In Experiment 1, participants were presented with a combination of spoken discourse and a sequence of cohesive gestures, which consistently located each of the two protagonists in two distinct locations in gesture space. Participants were asked to select an interpretation of the final sentence that best matched the preceding spoken and gestural contexts. Adults and 10-year-olds performed better than 5-year-olds, who were at chance level. In Experiment 2, another group of 5-year-olds was presented with the same stimuli as in Experiment 1, except that the actor showed hand-held pictures, instead of producing cohesive gestures. Unlike cohesive gestures, one set of pictures was self-explanatory and did not require integration with the concurrent speech to derive the referent. With these pictures, 5-year-olds performed nearly perfectly and their performance in the identifiable pictures was significantly better than those in the unidentifiable pictures. These results suggest that young children failed to integrate spoken discourse and cohesive use of space in gestures, because they cannot derive a referent of cohesive gestures from the local speech context.

Keywords:

• gesture
• speech
• discourse
• multimodal communication
• preschool children

Disclosure statement

No potential conflict of interest was reported by the authors.

Supplemental data

Supplemental data for this article can be accessed at http://dx.doi.org/10.1080/23273798.2015.1053814.

Notes

1. Similar type of gestures has been suggested by gesture researchers, such as abstract deixis/pointings (McNeill, 1992, 2005), referential gestures (Gullberg, 2006; Yoshioka, 2005) and entity gestures (Wilkin & Holler, 2011). The definitions of their gestures include the property of gesture that do not represent any existing space and that rather creates spaces and refer to locations that do not physically exist. However, they are different each other in terms of the repetitive quality, the usage in a discourse, the shape of gesture, type of speech accompanied with gesture.

2. We selected passages by conducting a pre-test. In the pre-test, we presented a written questionnaire, consisting of 20 candidate passages and forced choice questions, to 20 adult native speakers of Japanese who did not participate in the main experiment, and asked them to pick a correct answer from same three choices as the main experiments. Based on the result, we excluded passages in which more than 60% of the adults picked a particular choice. We finally selected 15 passages for the main experiment. The 15 messages are passages where adults picked the protagonist A choice (first-mentioned protagonist), protagonist B choice (second-mentioned protagonist) and both-protagonists choice in the forced choice question roughly equally often. For these 15 passages, the mean proportions of trials in which each choice was picked were similar across the three choices: M = .38 (SD = .22) for the protagonist A choice, M = .30 (SD = .22) for the protagonist B choice and M = .32 (SD = .28) for the both-choice. The probability of each choice being picked did not significantly differ from chance (.33) (protagonists A choice, t(19) = 1.15, ns; protagonists B choice, t(19) = .53, ns; both-choice, t(19) = .32, ns).

3. The honorific status (whether or not the referent of the subject should be respected) could be marked on the verb, but the items in this experiment did not have any honorific marking on the verbs.