Visual speech primes open-set recognition of spoken words

Abstract

Visual speech perception has become a topic of considerable interest to speech researchers. Previous research has demonstrated that perceivers neurally encode and use speech information from the visual modality, and this information has been found to facilitate spoken word recognition in tasks such as lexical decision (Kim, Davis, & Krins, 2004). In this paper, we used a cross-modality repetition priming paradigm with visual speech lexical primes and auditory lexical targets to explore the nature of this priming effect. First, we report that participants identified spoken words mixed with noise more accurately when the words were preceded by a visual speech prime of the same word compared with a control condition. Second, analyses of the responses indicated that both correct and incorrect responses were constrained by the visual speech information in the prime. These complementary results suggest that the visual speech primes have an effect on lexical access by increasing the likelihood that words with certain phonetic properties are selected. Third, we found that the cross-modality repetition priming effect was maintained even when visual and auditory signals came from different speakers, and thus different instances of the same lexical item. We discuss implications of these results for current theories of speech perception.

Keywords:

• Visual speech
• Audiovisual priming
• Word recognition
• Lexical access

Acknowledgements

This work was supported by NIH DC00012. The authors would like to thank Melissa Troyer for her assistance with this study, and Susannah Levi, Tessa Bent, Manuel Carreiras, and several anonymous reviewers for helpful comments on earlier drafts of this paper.

Notes

¹This was observed in pilot data for participants who saw trials with lexically consistent Visual Primes, lexically inconsistent dynamic speech primes and control static images. For these participants, there were no performance differences among these three groups. In addition, several pilot participants reported that they were not attending to the prime after realising it was sometimes not the target word. The inconsistent stimuli in the pilot were other words from the study that were randomly selected for display; thus, each word was CVC, but no other factors with respect to similarity were controlled.

²Unique responses are specific target-response combinations. For example, the target word ‘watch’ was produced accurately in the control condition by all subjects except for one, who wrote ‘what’. Thus, we considered this to be two unique responses to the word ‘watch’ – ‘watch’ and ‘what’. If another subject responded with ‘what’ to the target stimulus of ‘wash’, we called this an additional unique response as this is a different response-target pair.

³This type of theoretical approach posits that sensory information from the world is encoded in modality-specific representations, and that these modality-specific representations are either: (a) linked directly to one another (Massaro & Stork, 1998); or (b) linked to a separate ‘multimodal’ representation that integrates information from different sources (Hamilton et al., 2006; Skipper, Nusbaum, & Small, 2005). However, the difference between these proposals cannot be addressed by the research reported here.

⁴Several subjects were run on the same experiment with an ISI of 50 ms, and there were no differences between their data and those reported above. Given the relatively short ISI in these other participants, it is unlikely that these subjects were able to rehearse and decode the target word from the visual speech prime. However, it is worth noting that these similar patterns could arise from different loci, with the participants in the experiments reported here rehearsing and decoding the target words, and the participants who saw the stimuli with a 50 ms ISI exhibiting facilitatory effects of auditory processing from the visual speech signals (as in van Wassenhove, Grant, & Poeppel, 2005). We thank an anonymous reviewer for articulating this issue.