The costs of multimodal metaphors: comparing ERPs to figurative expressions in verbal and verbo-pictorial formats

ABSTRACT

Psycholinguistic research on metaphor has focused on verbal material. Yet, metaphors frequently occur in a multimodal format, blending words and pictures to convey meaning. Here we compared verbal and multimodal metaphors by using item pairs where stimulus one was always a word (e.g., language in the metaphorical conditions and river in the literal conditions) and stimulus two was either a word (bridge) or a picture (the image of a bridge). The two types of metaphors elicited a similar N400 effect compared to literal pairs; at later latencies, visual metaphors were associated with a more pronounced negativity compared to literal pictures, whereas no effect was observed in the verbal domain. These findings indicate that both visual and verbal metaphors recruit conceptual operations reflected in the N400, but for visual metaphors, elaboration lasts longer. This difference in time course was driven by the low number of alternative interpretations and their closedness, pointing to the costs, rather than the facilitation, of integrating visual signs at an abstract level of representation.

Acknowledgments

The EEG data were obtained under the agreement between Scuola Normale Superiore and Scuola Universitaria IUSS Pavia (2018-2020) for the realization of experiments on figurative language, and were collected at the Laboratorio SMART, section Elite, of the Scuola Normale Superiore in Pisa. In the final stages, this work received support from the European Research Council under the EU’s Horizon Europe programme, ERC Consolidator Grant “PROcessing MEtaphors: Neurochronometry, Acquisition and Decay, PROMENADE” (G.A. number: 101045733). The content of this article is the sole responsibility of the authors. The European Commission or its services cannot be held responsible for any use that may be made of the information it contains.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Author contribution

Valentina Bambini: Conceptualization, Methodology, Writing – original draft, Writing –review & editing, Supervision, Project administration, Funding acquisition. Giacomo Ranieri: Methodology, Software, Investigation. Luca Bischetti: Methodology, Formal analysis. Biagio Scalingi: Software, Investigation, Visualization. Chiara Bertini: Software, Investigation. Irene Ricci: Software, Investigation. Walter Schaeken: Conceptualization, Methodology. Paolo Canal: Conceptualization, Methodology, Formal analysis, Data curation, Writing – original draft, Writing – review & editing, Supervision, Visualization.

Notes

1. As an example of an open metaphorical ad, Ketelaar et al. (2012) discuss an Oil of Olaz ad containing a prominent picture of a crow holding crutches, accompanied by a pot of Oil of Olaz. The intended message is “Oil of Olaz makes crow’s feet disappear” (makes laughing wrinkles disappear). Yet, the study showed that very few participants in the study extracted the intended interpretations: some were not able to generate any interpretation at all, and most of them created several alternative interpretations, such as: “Oil of Olaz supports in maintaining a young skin,” “Oil of Olaz helps birds after oil disasters,” “Old disabled people have to use Oil of Olaz to get on their feet again,” “If you don’t use Oil of Olaz you will become an ugly old crow,” and “People fight for Oil of Olaz just as this blackbird did.”

2. The notion of interpretive diversity has been used in psycholinguistics to refer to the amount and distribution of interpretation (Reid et al., 2020) and is derived from the concept of entropy and calculated using the properties elicited by the metaphors (Roncero & de Almeida, 2015; Utsumi, 2005). Although interpretive diversity is similar to the notion of semantic distance among the interpretations that we used here, it cannot be equated with it (as it includes a measure of the distribution of the interpretations, which is not considered here) and neither with the broader notion of “openness,” which is rather an umbrella term capturing multiplicity, strength as well as availability of interpretations (see Introduction).

3. For this pair, rating values in the verbal vs. multimodal conditions were the following: number of interpretations = 2.32 vs. 1.5; strength of interpretation = 5.54 vs. 6.29; semantic distance between interpretations = 0.73 vs. 0.66.

Additional information

Funding

This work was supported by the European Research Council.