N400 Brain Responses to Spoken Phrases Paired with Photographs of Scenes: Implications for Visual Scene Displays in AAC Systems

Abstract

Augmentative and alternative communication (AAC) systems are often implemented for individuals whose speech cannot meet their full communication needs. One type of aided display is called a Visual Scene Display (VSD). VSDs consist of integrated scenes (such as photographs) in which language concepts are embedded. Often, the representations of concepts on VSDs are perceptually similar to their referents. Given this physical resemblance, one may ask how well VSDs support development of symbolic functioning. We used brain imaging techniques to examine whether matches and mismatches between the content of spoken messages and photographic images of scenes evoke neural activity similar to activity that occurs to spoken or written words. Electroencephalography (EEG) was recorded from 15 college students who were shown photographs paired with spoken phrases that were either matched or mismatched to the concepts embedded within each photograph. Of interest was the N400 component, a negative deflecting wave 400 ms post-stimulus that is considered to be an index of semantic functioning. An N400 response in the mismatched condition (but not the matched) would replicate brain responses to traditional linguistic symbols. An N400 was found, exclusively in the mismatched condition, suggesting that mismatches between spoken messages and VSD-type representations set the stage for the N400 in ways similar to traditional linguistic symbols.

Keywords::

• Electrophysiology
• Symbols
• N400
• Augmentative and alternative communication

Notes

Acknowledgements

We thank the willing volunteers for their time, and Michael Wenger for insights into the conceptualization of the research methods. Thanks also to Teresa Mitchell for her generous comments on this paper.

Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the article.

This research was supported by a Level 1 Award from the Social Sciences Research Institute of Penn State University. The first author is co-funded by the Children, Youth, and Families Consortium at Penn State.

Notes

1. Garage Band software (Version 2.0.1), is by Apple Inc., Cupertino, California.

2. The HydroCel Geodesic Sensor Net is a product of Electrical Geodesics, Inc. (EGI).