Neural systems involved in processing novel linguistic constructions and their visual referents

ABSTRACT

In language, abstract phrasal patterns provide an important source of meaning, but little is known about whether or how such constructions are used to predict upcoming visual scenes. Findings from two functional magnetic resonance imaging studies indicate that initial exposure to a novel construction allows its semantics to be used for such predictions. Specifically, greater activity in the ventral striatum, a region sensitive to prediction errors, was linked to worse overall comprehension of a novel construction. Moreover, activity in the occipital cortex was attenuated when a visual event could be inferred from a learned construction, which may reflect predictive coding of the event. These effects disappeared when predictions were unlikely: that is, when phrases provided no additional information about visual events. These findings support the idea that learners create and evaluate predictions about new instances during comprehension of novel linguistic constructions.

KEYWORDS:

• Language processing
• fMRI
• prediction
• visual scenes

Acknowledgements

The authors would like to thank Art Glenberg, Clarice Robenalt, the two anonymous reviewers, and the guest editor for this volume, Olaf Hauk, for their helpful comments on an earlier draft, and Jessica Hao for assistance with data collection and editing.

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes

1. Note that the negative relationship between ventral striatum activity and test performance may initially seem counterintuitive, as the size of a prediction error is related to the amount of ensuing learning (e.g. Rescorla & Wagner, 1972). However, remember that we examined brain activity only after the first two tokens of each construction had been witnessed during exposure, which allowed initial learning to take place. We thus interpret activity in the ventral striatum for later tokens as indicative of continued errors, that is, as evidence of insufficient knowledge of the construction for accurate predictions to be generated. The negative relationship between this activity and test performance is consistent with such an interpretation.

Additional information

Funding

This study was supported by NIH [grant number R01EY021755] to NTB and a fellowship from the Einstein Foundation of Berlin to AEG.