Hand Motion Oscillatory Gestures and Multimodal Discourse Analysis

Abstract

Gesture and speech are part of a single human language system. They are coexpressive and complementary channels in the act of speaking. Whereas speech carries the major load of symbolic presentation, gesture provides the imagistic content. Proceeding from the established cotemporality of gesture and speech, our work on oscillatory gestures and multimodal discourse is discussed. Our new techniques of analyzing hand gestures in the frequency domain are described. By tracking an individual's hands during a speech, hand motion trajectory signals are extracted from real video datasets. Our wavelet-based approach in gestural oscillation extraction is presented as frequency ridges in a frequency-time space. Wavelet ridges are extracted from responses of wavelet analysis. These wavelet ridges are employed to characterize frequency properties of hand motion trajectory signals. Hand motion oscillatory gestures can be extracted from these frequency properties. The potential of such computational cross-modal language analysis is motivated by performing a microanalysis of 2 video datasets. In the first dataset, a participant describes her living space to an interlocutor. In the second, a participant describes her action plan to an interlocutor. The ability of our algorithm to extract gestural oscillations is demonstrated, and the way that oscillatory gestures reveal portions of the discourse structure is shown.