The contribution of audiovisual speech to lexical-semantic processing in natural spoken sentences

References

• Alsius, A., Möttönen, R., Sams, M. E., Soto-Faraco, S., & Tiippana, K. (2014). Effect of attentional load on audiovisual speech perception: Evidence from ERPs. Fontiers in Psychology, 5, e727. doi: 10.3389/fpsyg.2014.00727
   PubMed Web of Science ®Google Scholar
• Altmann, G. T., & Kamide, Y. (1999). Incremental interpretation at verbs: Restricting the domain of subsequent reference. Cognition, 73, 247–264. doi: 10.1016/S0010-0277(99)00059-1
   PubMed Web of Science ®Google Scholar
• Arnal, L. H., Morillon, B., Kell, C. A., & Giraud, A.-L. (2009). Dual neural routing of visual facilitation in speech processing. The Journal of Neuroscience, 29, 13445–13453. doi: 10.1523/JNEUROSCI.3194-09.2009
   PubMed Web of Science ®Google Scholar
• Baart, M. (2016). Quantifying lip-read-induced suppression and facilitation of the auditory N1 and P2 reveals peak enhancements and delays. Psychophysiology, 53, 1295–1306. doi: 10.1111/psyp.12683
   PubMed Web of Science ®Google Scholar
• Baart, M., & Samuel, A. G. (2015). Turning a blind eye to the lexicon: ERPs show no cross-talk between lip-read and lexical context during speech sound processing. Journal of Memory and Language, 85, 42–59. doi: 10.1016/j.jml.2015.06.008
   Web of Science ®Google Scholar
• Baart, M., Stekelenburg, J. J., & Vroomen, J. (2016). Electrophysiological evidence for speech-specific audiovisual integration. Neuropsychologia, 53, 115–121. doi: 10.1016/j.neuropsychologia.2013.11.011
   Web of Science ®Google Scholar
• Badin, P., Tarabalka, Y., Elisei, F., & Bailly, G. (2010). Can you ‘read’ tongue movements? Evaluation of the contribution of tongue display to speech understanding. Speech Communication, 52, 493–503. doi: 10.1016/j.specom.2010.03.002
   Web of Science ®Google Scholar
• Ball, F., Michels, L. E., Thiele, C., & Noesselt, T. (2018). The role of multisensory interplay in enabling temporal expectations. Cognition, 170, 130–146. doi: 10.1016/j.cognition.2017.09.015
   PubMed Web of Science ®Google Scholar
• Basirat, A., Brunellière, A., & Hartsuiker, R. (2018). The role of audiovisual speech in the early stages of lexical processing as revealed by ERP word repetition effect. Language Learning, 68, 80–101. doi: 10.1111/lang.12265
   Web of Science ®Google Scholar
• Beauchamp, M. S., Argall, B. D., Bodurka, J., Duyn, J. H., & Martin, A. (2004). Unraveling multisensory integration: Patchy organization within human STS multisensory cortex. Nature Neuroscience, 7, 1190–1192. doi: 10.1038/nn1333
   PubMed Web of Science ®Google Scholar
• Benoit, C., Guiard-Marigny, T., Le Goff, B., & Adjoudani, A. (1996). Which components of the face do humans and machines best speechread? In D. G. Stork & M. E. Hennecke (Eds.), Speechreading by Humans and Machines (pp. 315–328). Berlin: NATO-ASI Series 150 Springer.
   Google Scholar
• Benoit, C., Lallouache, T., Mohamadi, T., & Abry, C. (1994). A set of French visemes for visual speech synthesis. Les Cahiers de l’ICP, Research Report, 3, 113–129.
   Google Scholar
• Besle, J., Fort, A., Delpuech, C., & Giard, M. H. (2004). Bimodal speech: Early suppressive visual effects in human auditory cortex. European Journal of Neuroscience, 20, 2225–2234. doi: 10.1111/j.1460-9568.2004.03670.x
   PubMed Web of Science ®Google Scholar
• Besle, J., Fort, A., & Giard, M. H. (2004). Interest and validity of the additive model in electrophysiological studies of multisensory interactions. Cognitive Processing, 5, 189–192. doi: 10.1007/s10339-004-0026-y
   Google Scholar
• Boersma, P., & Weenink, D. (2011). Praat: doing phonetics by computer [Computer program]. Version 3.4, retrieved 2 Jan 2011 from http://www.praat.org/
   Google Scholar
• Brunellière, A., Sánchez-García, C., Ikumi, N., & Soto-Faraco, S. (2013). Visual information constrains early and late stages of spoken-word recognition in sentence context. International Journal of Psychophysiology, 89, 136–147. doi: 10.1016/j.ijpsycho.2013.06.016
   PubMed Web of Science ®Google Scholar
• Brunellière, A., & Soto-Faraco, S. (2013). The speakers’ accent shapes the listeners’ phonological predictions during speech perception. Brain and Language, 125, 82–93. doi: 10.1016/j.bandl.2013.01.007
   PubMed Web of Science ®Google Scholar
• Brunellière, A., & Soto-Faraco, S. (2015). The interplay between semantic and phonological constraints during spoken-word comprehension. Psychophysiology, 52, 46–58. doi: 10.1111/psyp.12285
   PubMed Web of Science ®Google Scholar
• Buchwald, A. B., Winters, S. J., & Pisoni, D. B. (2009). Visual speech primes open-set recognition of spoken words. Language and Cognitive Processes, 24, 580–610. doi: 10.1080/01690960802536357
   PubMed Web of Science ®Google Scholar
• Calvert, G. A., Campbell, R., & Brammer, M. J. (2000). Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Current Biology, 10, 649–657. doi: 10.1016/S0960-9822(00)00513-3
   PubMed Web of Science ®Google Scholar
• Chandrasekaran, C., Trubanova, A., Stillittano, S., Caplier, A., & Ghazanfar, A. A. (2009). The natural statistics of audiovisual speech. PLoS Computing Biology, 5, e1000436. doi: 10.1371/journal.pcbi.1000436
   PubMed Web of Science ®Google Scholar
• Connolly, J. F., & Phillips, N. A. (1994). Event-related potential components reflect phonological and semantic processing of the terminal word of spoken sentences. Journal of Cognitive Neuroscience, 6, 256–266. doi: 10.1162/jocn.1994.6.3.256
   PubMed Web of Science ®Google Scholar
• Connolly, J. F., Phillips, N. A., Stewart, S. H., & Brake, W. G. (1992). Event-related potential sensitivity to acoustic and semantic properties of terminal words in sentences. Brain and Language, 43, 1–18. doi: 10.1016/0093-934X(92)90018-A
   PubMed Web of Science ®Google Scholar
• Connolly, J. F., Stewart, S. H., & Phillips, N. A. (1990). The effects of processing requirements on neurophysiological responses to spoken sentences. Brain and Language, 39, 302–318. doi: 10.1016/0093-934X(90)90016-A
   PubMed Web of Science ®Google Scholar
• Dodd, B., Oerlemens, M., & Robinson, R. (1989). Cross-modal effects in repetition priming: A comparison of lip-read graphic and heard stimuli. Visible Language, 22, 59–77.
   Google Scholar
• Fisher, C. G. (1968). Confusions among visually perceived consonants. Journal of Speech and Hearing Research, 11, 796–804. doi: 10.1044/jshr.1104.796
   PubMedGoogle Scholar
• Fort, M., Kandel, S., Chipot, J., Savariaux, C., Granjon, L., & Spinelli, E. (2013). Seeing the initial articulatory gestures of a word triggers lexical access. Language and Cognitive Processes, 28, 1207–1223. doi: 10.1080/01690965.2012.701758
   Web of Science ®Google Scholar
• Foucart, A., Ruiz-Tada, E., & Costa, A. (2015). How do you know I was about to say “book”? Anticipation processes affect speech processing and lexical recognition. Language, Cognition and Neuroscience, 30, 768–780. doi: 10.1080/23273798.2015.1016047
   Web of Science ®Google Scholar
• Fridriksson, J., Moss, J., Davis, B., Baylis, G. C., Bonilha, L., & Rorden, C. (2008). Motor speech perception modulates the cortical language areas. NeuroImage, 41, 605–613. doi: 10.1016/j.neuroimage.2008.02.046
   PubMed Web of Science ®Google Scholar
• Friston, K., & Kiebel, S. (2009). Cortical circuits for perceptual inference. Neural Networks, 22, 1093–1104. doi: 10.1016/j.neunet.2009.07.023
   PubMed Web of Science ®Google Scholar
• Gaskell, M. G., & Marslen-Wilson, W. D. (1997). Integrating form and meaning: A distributed model of speech perception. Language and Cognitive Processes, 12, 613–656. doi: 10.1080/016909697386646
   Web of Science ®Google Scholar
• Goolkasian, P., & Foos, P. W. (2005). Bimodal format effects in working memory. American Journal of Psychology, 118, 61–77.
   PubMed Web of Science ®Google Scholar
• Grant, K. W., & Seitz, P. F. (1998). Measures of auditory-visual integration in nonsense syllables and sentences. Journal of the Acoustical Society of America, 104, 2438–2450. doi: 10.1121/1.423751
   PubMed Web of Science ®Google Scholar
• Greenhouse, S. W., & Geisser, S. (1959). On methods in the analysis of profile data. Psychometrika, 24, 95–111. doi: 10.1007/BF02289823
   Web of Science ®Google Scholar
• Hagoort, P., & Brown, C. M. (2000). ERP effects of listening to speech: Semantic ERP effects. Neuropsychologia, 38, 1518–1530. doi: 10.1016/S0028-3932(00)00052-X
   PubMed Web of Science ®Google Scholar
• Hall, D. A., Fussell, C., & Summerfield, A. Q. (2005). Reading fluent speech from talking faces: Typical brain networks and individual differences. Journal of Cognitive Neuroscience, 17, 939–953. doi: 10.1162/0898929054021175
   PubMed Web of Science ®Google Scholar
• Hickock, G., & Poeppel, D. (2007). The cortical organization of speech processing. Nature Reviews Neuroscience, 8, 393–402. doi:10.1016/j.jcomdis.2012.06.004 doi: 10.1038/nrn2113
   PubMed Web of Science ®Google Scholar
• Istria, M., Nicolas-Jeantoux, C., & Tamboise, J. (1982). Manuel de lecture labiale. Exercices d'entraînement. Paris: Masson.
   Google Scholar
• Jesse, A., & Massaro, D. W. (2010). The temporal distribution of information in audiovisual spoken-word identification. Attention, Perception, Psychophysics, 72, 209–225. doi: 10.3758/APP.72.1.209
   PubMed Web of Science ®Google Scholar
• Kaiser, E., & Trueswell, J. C. (2004). The role of discourse context in the processing of a flexible word-order language. Cognition, 94, 113–147. doi: 10.1016/j.cognition.2004.01.002
   PubMed Web of Science ®Google Scholar
• Kamide, Y., Altmann, G. T., & Haywood, S. L. (2003). The time-course of prediction in incremental sentence processing: Evidence from anticipatory eye movements. Journal of Memory and Language, 49, 133–156. doi: 10.1016/S0749-596X(03)00023-8
   Web of Science ®Google Scholar
• Kim, J., Davis, C., & Krins, P. (2004). A modal processing of visual speech as revealed by priming. Cognition, 93, B39–B47. doi: 10.1016/j.cognition.2003.11.003
   PubMed Web of Science ®Google Scholar
• Kim, A., & Lai, V. (2012). Rapid interactions between lexical semantic and word form analysis during word recognition in context: Evidence from ERPs. Journal of Cognitive Neuroscience, 24, 1104–1112. doi: 10.1162/jocn_a_00148
   PubMed Web of Science ®Google Scholar
• Klucharev, V., Möttönen, R., & Sams, M. (2003). Electrophysiological indicators of phonetic and non-phonetic multisensory interactions during audiovisual speech perception. Brain Cognitive Research, 18, 65–75. doi: 10.1016/j.cogbrainres.2003.09.004
   PubMedGoogle Scholar
• Kutas, M., & Federmeier, K. D. (2000). Electrophysiology reveals semantic memory use in language comprehension. Trends in Cognitive Sciences, 12, 463–470. doi: 10.1016/S1364-6613(00)01560-6
   Google Scholar
• Lange, K. (2012). The N1 effect of temporal attention is independent of sound location and intensity: Implications for possible mechanisms of temporal attention. Psychophysiology, 49, 1468–1480. doi: 10.1111/j.1469-8986.2012.01460.x
   PubMed Web of Science ®Google Scholar
• Lange, K., & Schnuerch, R. (2014). Challenging perceptual tasks require more attention: The influence of task difficulty on the N1 effect of temporal orienting. Brain and Cognition, 84, 153–163. doi: 10.1016/j.bandc.2013.12.001
   PubMed Web of Science ®Google Scholar
• Lau, E. F., Phillips, C., & Poeppel, D. (2008). A cortical network for semantics: (de)constructing the N400. Nature Reviews Neuroscience, 9, 920–933. doi: 10.1038/nrn2532
   PubMed Web of Science ®Google Scholar
• Ma, W. J., Zhou, X., Ross, L. A., Foxe, J. J., & Parra, L. C. (2009). Lip-reading aids word recognition most in moderate noise: A Bayesian explanation using high-dimensional feature space. PLoS One, 4, e4638. doi: 10.1371/journal.pone.0004638
   PubMed Web of Science ®Google Scholar
• MacLeod, A., & Summerfield, Q. (1987). Quantifying the contribution of vision to speech perception in noise. British Journal of Audiology, 21, 131–141. doi: 10.3109/03005368709077786
   PubMedGoogle Scholar
• Marslen-Wilson, W. D. (1984). Function and process in spoken word recognition: A tutorial review. In H. Bouma, & D. G. Bouwhis (Eds.), Attention and performance X: Control of language processes (pp. 125–150). Hillsdale, NJ: Erlbaum.
   Google Scholar
• Massaro, D. W. (1998). Perceiving talking faces: From speech perception to a behavioral principle. Cambridge, MA: MIT Press.
   Google Scholar
• Mattys, S. L., Bernstein, L. E., & Auer, E. T. (2002). Stimulus-based lexical distinctiveness as a general word recognition mechanism. Perception and Psychophysics, 64, 667–679. doi: 10.3758/BF03194734
   PubMed Web of Science ®Google Scholar
• McClelland, J. L., & Elman, J. L. (1986). The trace model of speech perception. Cognitive Psychology, 18, 1–86. doi: 10.1016/0010-0285(86)90015-0
   PubMed Web of Science ®Google Scholar
• McGrath, M. (1985). An examination of cues for visual and audiovisual speech perception using natural and computer generated faces. PhD Dissertation. University of Nottingham, United Kingdom.
   Google Scholar
• Navarra, J., & Soto-Faraco, S. (2007). Hearing lips in a second language: Visual articulatory information enables the perception of second language sounds. Psychological Research, 71, 4–12. doi: 10.1007/s00426-005-0031-5
   PubMed Web of Science ®Google Scholar
• Neville, H. J., Kutas, M., Chesney, G., & Schmidt, A. L. (1986). Event-related brain potentials during initial encoding and recognition memory of congruous and incongruous words. Journal of Memory and Language, 25, 75–92. doi : 10.1016/0749-596X(86)90022-7
   Web of Science ®Google Scholar
• New, B., Pallier, C., Ferrand, L., & Matos, R. (2001). Une base de données lexicales du français contemporain sur internet: LEXIQUE. L’Année Psychologique, 101, 447–462. doi: 10.3406/psy.2001.1341
   Web of Science ®Google Scholar
• Oldfield, R. C. (1971). The assessment and analysis of handedness: The Edinburgh inventory. Neuropsychologia, 9, 97–113. doi: 10.1016/0028-3932(71)90067-4
   PubMed Web of Science ®Google Scholar
• Otten, M., Nieuwland, M. S., & van Berkum, J. A. (2007). Great expectations: Specific lexical anticipation influences the processing of spoken language. BMC Neuroscience, 8, 89. doi: 10.1186/1471-2202-8-89
   PubMed Web of Science ®Google Scholar
• Peelle, J. E., & Sommers, M. S. (2015). Prediction and constraint in audiovisual speech perception. Cortex, 68, 169–181. doi: 10.1016/j.cortex.2015.03.006
   PubMed Web of Science ®Google Scholar
• Penolazzi, B., Hauk, O., & Pulvermüller, F. (2007). Early semantic context integration and lexical access as revealed by event-related brain potentials. Biological Psychology, 74, 374–388. doi: 10.1016/j.biopsycho.2006.09.008
   PubMed Web of Science ®Google Scholar
• Perrin, F., Pernier, J., Bertrand, O., Giard, M.-H., & Echallier, J. F. (1987). Mapping of scalp potentials by surface spline interpolation. Electroencephalography and Clinical Neurophysiology, 66, 75–81. doi: 10.1016/0013-4694(87)90141-6
   PubMedGoogle Scholar
• Pichora-Fuller, M. K. (1996). Working memory and speechreading. In D. G. Stork, & M. E. Hennecke (Eds.), Speechreading by humans and machines: Models, systems, and applications (pp. 257–274). Berlin: Springer.
   Google Scholar
• Pilling, M. (2009). Auditory event-related potentials (ERPs) in audiovisual speech perception. Journal of Speech, Language and Hearing Research, 52, 1073–1081. doi:10.1044/1092-4388(2009/07-0276 doi: 10.1044/1092-4388(2009/07-0276)
   PubMed Web of Science ®Google Scholar
• Power, A. J., Foxe, J. J., Forde, E.-J., Reilly, R. B., & Lalor, E. C. (2012). At what time is the cocktail party? A late locus of selective attention to natural speech. European Journal of Neuroscience, 35, 1497–1503. doi: 10.1111/j.1460-9568.2012.08060.x
   PubMed Web of Science ®Google Scholar
• Ross, L. A., Saint-Amour, D., Leavitt, V. M., Javitt, D. C., & Foxe, J. J. (2007). Do you see what I am saying? Exploring visual enhancement of speech comprehension in noisy environments. Cerebral Cortex, 17, 1147–1153. doi: 10.1093/cercor/bhl024
   PubMed Web of Science ®Google Scholar
• Samuel, A. G., & Lieblich, J. (2014). Visual speech acts differently than lexical context in supporting speech perception. Journal of Experimental Psychology: Human Perception and Performance, 40, 1479–1490. doi: 10.1037/a0036656
   PubMed Web of Science ®Google Scholar
• Schwartz, J.-L., & Savariaux, C. (2014). No, there is no 150 ms lead of visual speech on auditory speech, but a range of audiovisual asynchronies varying from small audio lead to large audio lag. PLoS Computational Biology, 10, e1003743. doi: 10.1371/journal.pcbi.1003743
   PubMed Web of Science ®Google Scholar
• Sekiyama, K., Kanno, I., Miura, S., & Sugita, Y. (2003). Auditory-visual speech perception examined by fMRI and PET. Neuroscience Research, 47, 277–287. doi: 10.1016/S0168-0102(03)00214-1
   PubMed Web of Science ®Google Scholar
• Skipper, J. I., Nusbaum, H., & Small, S. L. (2005). Listening to talking faces: Motor cortical activation during speech perception. NeuroImage, 25, 76–89. doi: 10.1016/j.neuroimage.2004.11.006
   PubMed Web of Science ®Google Scholar
• Stekelenburg, J. J., & Vroomen, J. (2007). Neural correlates of multisensory integration of ecologically valid audiovisual events. Journal of Cognitive Neuroscience, 19, 1964–1973. doi: 10.1162/jocn.2007.19.12.1964
   PubMed Web of Science ®Google Scholar
• Sumby, W. H., & Pollack, I. (1954). Visual contribution to speech intelligibility in noise. The Journal of the Acoustical Society of America, 26, 212–215. doi: 10.1121/1.1907309
   Web of Science ®Google Scholar
• Summerfield, Q., & McGrath, M. (1984). Detection and resolution of audio-visual incompatibility in the perception of vowels. Quarterly Journal of Experimental Psychology, 36, 51–74. doi: 10.1080/14640748408401503
   Web of Science ®Google Scholar
• Teder-Salejarvi, W. A., McDonald, J. J., Di Russo, F., & Hillyard, S. A. (2002). An analysis of audio-visual crossmodal integration by means of event-related potential (ERP) recordings. Cognitive Brain Research, 14, 106–114. doi :10.1016/S0926-6410(02)00065-4
   PubMedGoogle Scholar
• Thompson, V., & Paivio, A. (1994). Memory for pictures and sounds: Independence of auditory and visual codes. Canadian Journal of Experimental Psychology, 48, 380–398. doi: 10.1037/1196-1961.48.3.380
   PubMed Web of Science ®Google Scholar
• Tye-Murray, N., Sommers, M. S., & Spehar, B. (2007). Auditory and visual lexical neighborhoods in audiovisual speech perception. Trends in Amplification, 11, 233–241. doi: 10.1177/1084713807307409
   PubMedGoogle Scholar
• van Berkum, J. J., Brown, C. M., Zwitserlood, P., Kooijman, V., & Hagoort, P. (2005). Anticipating upcoming words in Discourse: Evidence from ERPs and reading times. Journal of Experimental Psychology: Learning, Memory, and Cognition, 31, 443–467. doi; 10.1037/0278-7393.31.3.443
   PubMed Web of Science ®Google Scholar
• van den Brink, D., Brown, C. M., & Hagoort, P. (2001). Electrophysiological evidence for early contextual influences during spoken-word recognition: N200 versus N400 effects. Journal of Cognitive Neuroscience, 13, 967–985. doi: 10.1162/089892901753165872
   PubMed Web of Science ®Google Scholar
• van den Brink, D., & Hagoort, P. (2004). The influence of semantic and syntactic context constraints on lexical selection and integration in spoken-word comprehension as revealed by ERPs. Journal of Cognitive Neuroscience, 16, 1068–1084. doi: 10.1162/0898929041502670
   PubMed Web of Science ®Google Scholar
• Van Engen, K. J., Xie, Z., & Chandrasekaran, B. (2017). Audiovisual sentence recognition not predicted by susceptibility to the McGurk effect. Attention, Perception & Psychophysics, 79, 396–403. doi: 10.3758/s13414-016-1238-9
   PubMed Web of Science ®Google Scholar
• van Wassenhove, V., Grant, K. W., & Poeppel, D. (2005). Visual speech speeds up the neural processing of auditory speech. Proceedings of the National Academy of Sciences, 102, 1181–1186. doi: 10.1073/pnas.0408949102
   PubMed Web of Science ®Google Scholar
• Wicha, N. Y. Y., Bates, E. A., Moreno, E. M., & Kutas, M. (2003). Potato not Pope: Human brain potentials to gender expectation and agreement in Spanish spoken sentences. Neuroscience Letters, 346, 165–168. doi: 10.1016/S0304-3940(03)00599-8
   PubMed Web of Science ®Google Scholar
• Wright, T. M., Pelphrey, K. A., Allison, T., McKeown, M. J., & McCarthy, G. (2003). Polysensory interactions along lateral temporal regions evoked by audiovisual speech. Cerebral Cortex, 13, 1034–1043. doi: 10.1093/cercor/13.10.1034
   PubMed Web of Science ®Google Scholar