The interaction of visual and linguistic saliency during syntactic ambiguity resolution

Abstract

Psycholinguistic research using the visual world paradigm has shown that the processing of sentences is constrained by the visual context in which they occur. Recently, there has been growing interest in the interactions observed when both language and vision provide relevant information during sentence processing. In three visual world experiments on syntactic ambiguity resolution, we investigate how visual and linguistic information influence the interpretation of ambiguous sentences. We hypothesize that (1) visual and linguistic information both constrain which interpretation is pursued by the sentence processor, and (2) the two types of information act upon the interpretation of the sentence at different points during processing. In Experiment 1, we show that visual saliency is utilized to anticipate the upcoming arguments of a verb. In Experiment 2, we operationalize linguistic saliency using intonational breaks and demonstrate that these give prominence to linguistic referents. These results confirm prediction (1). In Experiment 3, we manipulate visual and linguistic saliency together and find that both types of information are used, but at different points in the sentence, to incrementally update its current interpretation. This finding is consistent with prediction (2). Overall, our results suggest an adaptive processing architecture in which different types of information are used when they become available, optimizing different aspects of situated language processing.

Keywords:

• Situated language comprehension
• Visual saliency
• Intonational breaks
• Adaptive constraint-based architecture
• Ambiguity resolution

European Research Council under award number 203427 “Synchronous Linguistic and Visual Processing” is gratefully acknowledged. At the time of publication, M.I.C. is Research Associate at the Department of Psychology, University of Lisbon.

Notes

¹In this paper, we will use small caps to indicate visual objects, and italics to refer to words and phrases.

²In the context of foreign accents, saliency is sometimes also regarded as an acoustic feature (Bradlow, Clopper, Smiljanic, & Walter, 2010).

³Though intonational information can also have pragmatic effects, or affect the truth-conditional meaning of a sentence (Jackendoff, 2002).

⁴We also tried only excluding fixations launched after the end of the ROI, and obtained exactly the same pattern of results.

⁵We also tested models with correlated random slopes, and found the same pattern of results.