Development of Early Multisensory Perception and Communication: From Environmental and Behavioral to Neural Signatures

Multidisciplinary research across the domains of cognitive development, speech, hearing, and educational sciences, field studies in cultural anthropology, and more recently, empirical research in artificial intelligence, computational modeling, and robotics have firmly established that caregiver-infant communication is a multisensory activity. Infants learn about communication and learn to communicate during ongoing interaction with caregivers in a multisensory world. These related domains of investigation, spanning four decades, have provided rich insights into the multisensory properties of caregivers’ facial and vocal expressions, various speech modulations, head and body movements and touch during discourse while naming objects and actions, all embedded in joint attention activities engaging infants to enhance their communicative abilities (Brand, Baldwin, & Ashburn, 2002; Gogate, Bahrick, & Watson, 2000; Gogate, Bolzani, & Betancourt, 2006; Gogate, Maganti, & Bahrick, 2015; Harris, Jones, & Grant, 1983; Messer, 1978; Rohlfing, Fritsch, Wrede, & Jungmann, 2006; Smith & Strader, 2014; Sullivan & Horowitz, 1983; Yu & Smith, 2012; Zukow-Goldring, 1990). At the same time, experimental (behavioral) research in developmental psychology has established that young infants prefer this multisensory (e.g., auditory-visual) communication over its unisensory (e.g., auditory or visual) components (e.g., Bahrick & Pickens, 1988; Patterson & Werker, 2003). These experimental studies have underscored that typically developing infants prefer temporally synchronous and spatially collocated faces and voices to asynchronous faces and voices during speech (e.g., Dodd, 1979; Hollich, Newman, & Juscyk, 2005; Kuhl & Meltzoff, 1984; Lewkowicz, 2002, 2010; Walker, 1982), as well as synchronous spoken words and moving referents to asynchronous ones (Gogate, 2010; Jesse & Johnson, 2012; Rader & Zukow-Goldring, 2015).

The neurophysiological correlates of multisensory (auditory-visual) caregiver-infant communication, however, have remained largely unexplored barring a few exceptions. For instance, only recently have researchers started to explore the neurophysiological components of multisensory face-voice perception in infancy (Grossman, 2010; Hyde, Jones, Flom, & Porter, 2011; Reynolds, Bahrick, Lickliter, & Guy, 2014), although earlier research suggested that intersensory face-voice processing was relegated to the left hemisphere (MacKain, Studdert-Kennedy, Spieker, & Stern, 1983). Furthermore, although some pioneering neurophysiological research has examined the beginnings of auditory-visual word-object mapping in infants (Asano et al., 2015; Friederich & Friederici, 2011, 2015; Junge, Cutler, & Hagoort, 2012; Molfese, 1989; Molfese, Morse, & Peters, 1990), no studies to date have examined the cortical regions involved in infants’ processing of specific multisensory properties, such as temporal synchrony processing during word-referent mapping when caregivers name objects and events, or infants’ processing of other levels of multisensory discourse. The neural signatures of multisensory communication across key participants, its correlates in caregivers and in infants during ongoing interaction, have received very little attention in empirical research. Although the methods for examining the neural signatures of multisensory communication have been available for decades, researchers have only recently acknowledged the importance of examining the development of these signatures in infants and their caregivers (Tanaka, Fukushima, Okanayo, & Myowa-Yamakoshi, 2014), as a means for understanding communicative development.

The present issue aims to highlight this largely unchartered territory of research to better understand the developmental origins of human multisensory communication. Taken together, the two review and four empirical articles emphasize the range and complexity of multisensory mechanisms underlying infant-caregiver dyadic and triadic exchanges in the real world, and multiple facets of their developmental and neurophysiological characteristics. Furthermore, recent theoretical perspectives on brain network development emphasize the need for elucidating the nature of the environment to better comprehend neurophysiological development (e.g., Byrge, Sporns, & Smith, 2014), because the brain, body and environment interact with one another on an ongoing basis, influence each other and are continuously being influenced by one another (e.g., James & Maouene, 2009; Maouene, Hidaka, & Smith, 2008). Thus, the set of papers included in this special issue examine neurophysiological, behavioral and environmental factors to better understand the nature of multisensory communication as a whole.

The present issue begins with a review by Hyde, Flom, and Porter of the neurophysiological and behavioral research to illustrate that infants’ face-voice processing during auditory-visual communication involves general principles of perceptual processing. Their main contention is that despite human communication being a multisensory experience, research and theorizing on infant perception, cognition, and learning related to face-voice perception has largely focused on unimodal or single sense modalities. The same contention holds for research on the neurophysiological underpinnings of auditory-visual communication in general.

Next, Gogate and Hollich review the neurophysiological and behavioral research on noun-object and verb-action mapping across auditory-visual sensory modalities in the early years. Once again, they emphasize the domain-general multisensory perceptual origins and neural underpinnings that give way to language-specific biases in verb and noun learning, eventually leading to children becoming more flexible with noun and verb learning that is less constrained by a noun- or a verb-bias.

The set of empirical papers begins with Borgström, Torkildsen and Lindgren who illustrate how visual perception of familiar objects facilitates toddlers’ neurophysiological responses (N400 amplitude) to pseudo word-referent incongruity. These findings speak to the dependency of language learning on domain-general visual shape perception and simultaneously attest to the multisensory nature of word learning. Next, Suanda, Smith and Yu examine the multisensory properties of caregivers’ extended discourse to toddlers during ongoing parent-child interactions, specifically illustrating several types of redundancies between caregivers’ gestures and their extended conversations as a means for highlighting meaning to toddlers. Complementing Suanda et al.’s empirical findings, Chang, de Barbaro, and Deák examine longitudinally the contingencies between infants’ behaviors with objects and maternal naming. They illustrate increasing redundancies between infants’ gestures, eye-gaze and maternal speech as infants’ develop from 4 to 9 months of age. Chang et al. demonstrate that with increasing infant age, caregivers name objects more often when infants vocalize, or manipulate objects, or gaze at them. The latter two empirical studies underscore the rich contexts that constitute ongoing caregiver-child multisensory communicative interactions, and highlight the ongoing brain-body-environment relations that characterize these interactions. Finally, Patten, Labban, Casenhiser, and Cotton examine the influence of one aspect of this rich context, the absence versus the presence of temporally synchronous faces during naming, to determine their influence on the perception of these linguistic stimuli by children on the autistic spectrum. This research speaks directly to the clinical relevance of multisensory perceptual mechanisms in translational research. Together, these articles underscore the domain-general multisensory nature of human communication in the early years and suggest further research to elucidate the embodied neural signatures of multisensory communication between infants, children, and their caregivers.

Author's Note

LG expresses her deepest appreciation to the authors for their invaluable contributions to this issue, and to the numerous reviewers who generously offered their time and effort to provide expert reviews.