Representing coordination and non-coordination in American Sign Language animations

Abstract

While strings and syntax trees are used by the Natural Language Processing community to represent the structure of spoken languages, these encodings are difficult to adapt to a signed language like American Sign Language (ASL). In particular, the multichannel nature of an ASL performance makes it difficult to encode in a linear single-channel string. This paper will introduce the Partition/Constitute (P/C) formalism, a new method of computationally representing a linguistic signal containing multiple channels. The formalism allows coordination and non-coordination relationships to be encoded between different portions of a signal. The P/C formalism will be compared to representations used in related research in gesture animation. The way in which P/C is used by this project to build an English-to-ASL machine translation system will also be discussed.

Keywords:

• American Sign Language
• Machine translation
• Accessibility technology for the deaf
• Multimodal generation
• Gesture generation
• Partition/Constitute formalism

Acknowledgements

This project is supported by the US National Science Foundation (NSF) Universal Access Program (Award #0520798 ‘SGER: Generating Animations of American Sign Language Classifier Predicates’) and an NSF Graduate Research Fellowship to this author. I would like to thank my advisors Dr. Mitch Marcus and Dr. Martha Palmer for their guidance, discussion, and revisions during the preparation of this work. I would also like to thank Dr. Norman Badler for his help in obtaining funding for this research.

Notes

¹There are some details about the timing of the left and right head movements which are not fully discussed in this paper. For instance, the first ‘left’ head movement in figure 2 may actually be an anticipatory movement which is needed to get the signer's head in the proper position to begin the ‘Negative Headshake’ performance. It may actually be the first ‘right’ movement that is coordinated with the start of the sign NOT.

²The Partition/Constitute formalism (presented later) can be formulated as a special 2D instance of these grammars. Constituting rules could use constraints to enforce that their sub-nodes are ordered and adjacent; partitioning rules could enforce sub-nodes to cover their parent's channels in a non-overlapping way.