260
Views
1
CrossRef citations to date
0
Altmetric
Articles

The costs of multimodal metaphors: comparing ERPs to figurative expressions in verbal and verbo-pictorial formats

, , , , , , & show all

References

  • Al-Azary, H., & Buchanan, L. (2017). Novel metaphor comprehension: Semantic neighbourhood density interacts with concreteness. Memory & Cognition, 45(2), 296–307. https://doi.org/10.3758/s13421-016-0650-7
  • Al-Azary, H., & Katz, A. N. (2021). Do metaphorical sharks bite? Simulation and abstraction in metaphor processing. Memory & Cognition, 49(3), 557–570. https://doi.org/10.3758/s13421-020-01109-2
  • Arcara, G., Lacaita, G., Mattaloni, E., Passarini, L., Mondini, S., Benincà, P., & Semenza, C. (2012). Is ‘Hit and run’ a single word? The processing of irreversible binomials in neglect dyslexia. Frontiers in Psychology, 3, 11. https://doi.org/10.3389/fpsyg.2012.00011
  • Baggio, G., van Lambalgen, M., & Hagoort, P. (2008). Computing and recomputing discourse models: An ERP study. Journal of Memory and Language, 52(1), 36–53. https://doi.org/10.1016/j.jml.2008.02.005
  • Bambini, V., Bertini, C., Schaeken, W., Stella, A., & Di Russo, F. (2016). Disentangling metaphor from context: An ERP study. Frontiers in Psychology, 7, 559. https://doi.org/10.3389/fpsyg.2016.00559
  • Bambini, V., Bott, L., & Schumacher, P. B. (2021). It is not always a matter of time: Addressing the costs of metaphor and metonymy through a speed-accuracy trade-off study. Canadian Journal of Experimental Psychology/Revue Canadienne de Psychologie Expérimentale, 75(2), 189–196. https://doi.org/10.1037/cep0000256
  • Bambini, V., Canal, P., Resta, D., & Grimaldi, M. (2019). Time course and neurophysiological underpinnings of metaphor in literary context. Discourse Processes, 56(1), 77–97. https://doi.org/10.1080/0163853X.2017.1401876
  • Bambini, V., Ghio, M., Moro, A., & Schumacher, P. B. (2013). Differentiating among pragmatic uses of words through timed sensicality judgments. Frontiers in Psychology, 4, 938. https://doi.org/10.3389/fpsyg.2013.00938
  • Bambini, V., & Resta, D. (2012). Metaphor and experimental pragmatics: When theory meets empirical investigation. Humanamente, 5(23), 37–60.
  • Bambini, V., Resta, D., & Grimaldi, M. (2014). A dataset of metaphors from the Italian literature: Exploring psycholinguistic variables and the role of context. PLoS ONE, 9(9), e105634. https://doi.org/10.1371/journal.pone.0105634
  • Bates, D., Kliegl, R., Vasishth, S., & Baayen, H. (2015). Parsimonious mixed models. arXiv:1506.04967.
  • Bates, D., Mächler, M., Bolker, B., & Walker, S. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 1–48. https://doi.org/10.18637/jss.v067.i01
  • Bolognesi, M., van den Heerik, R., van den Berg, E. (2018). VisMet 1.0 an online corpus of visual metaphors, and G. J. Steen (Ed.), Visual metaphor (pp. 89–114). John Benjamins. https://doi.org/10.1075/celcr.18.04bol
  • Bolognesi, M., & Werkmann Horvat, A. (2023). The metaphor compass. Directions for metaphor research in language, cognition, communication, and creativity. Routledge.
  • Brodeur, M. B., Guérard, K., & Bouras, M. (2014). Bank of Standardized Stimuli (BOSS) phase ii: 930 new normative photos. PLoS ONE, 9(9), e106953. https://doi.org/10.1371/JOURNAL.PONE.0106953
  • Brouwer, H., & Hoeks, J. C. J. (2013). A time and place for language comprehension: Mapping the N400 and the P600 to a minimal cortical network. Frontiers in Human Neuroscience, 7, 758. https://doi.org/10.3389/fnhum.2013.00758
  • Campbell, S. J., & Raney, G. E. (2016). A 25-year replication of Katz et al.’s (1988) metaphor norms. Behavior Research Methods, 48(1), 330–340. https://doi.org/10.3758/s13428-015-0575-2
  • Canal, P., Bambini, V. (2023). Pragmatics electrified. In M. Grimaldi, Y. Shtyrov, & E. Brattico (Eds.), Language electrified. Principles, methods, and future perspectives of investigation (pp. 583–612). Humana. https://doi.org/10.1007/978-1-0716-3263-5_18
  • Canal, P., Bischetti, L., Bertini, C., Ricci, I., Lecce, S., & Bambini, V. (2022). N400 differences between physical and mental metaphors: The role of theories of mind. Brain and Cognition, 161, 105879. https://doi.org/10.1016/j.bandc.2022.105879
  • Canal, P., Bischetti, L., Di Paola, S., Bertini, C., Ricci, I., & Bambini, V. (2019). ‘Honey, shall I change the baby? – Well done, choose another one’: ERP and time-frequency correlates of humor processing. Brain and Cognition, 132, 41–55. https://doi.org/10.1016/J.BANDC.2019.02.001
  • Cao, S., Wang, Y., Chen, H., & Wang, H. (2018). The N1-N2-LPC pattern in processing advertising pictorial metaphors: An ERP study. Frontiers in Psychology, 9, 2566. https://doi.org/10.3389/fpsyg.2018.02566
  • Carston, R. (2010). XIII—Metaphor: Ad hoc concepts, literal meaning and mental images. Proceedings of the Aristotelian Society, 110(3pt3), 295–321. https://doi.org/10.1111/j.1467-9264.2010.00288.x
  • Carston, R. (2018). Figurative language, mental imagery, and pragmatics. Metaphor and Symbol, 33(3), 198–217. https://doi.org/10.1080/10926488.2018.1481257
  • Cohn, N. (2010). Extra! Extra! Semantics in comics!: The conceptual structure of Chicago Tribune advertisements. Journal of Pragmatics, 42(11), 3138–3146. https://doi.org/10.1016/j.pragma.2010.04.016
  • Cohn, N. (2020). Your brain on comics: A cognitive model of visual narrative comprehension. Topics in Cognitive Science, 12(1), 352–386. https://doi.org/10.1111/tops.12421
  • Cohn, N., & Foulsham, T. (2022). Meaning above (and in) the head: Combinatorial visual morphology from comics and emoji. Memory & Cognition, 50(7), 1381–1398. https://doi.org/10.3758/s13421-022-01294-2
  • Cohn, N., & Kutas, M. (2015). Getting a cue before getting a clue: Event-related potentials to inference in visual narrative comprehension. Neuropsychologia, 77, 267–278. https://doi.org/10.1016/j.neuropsychologia.2015.08.026
  • De Grauwe, S., Swain, A., Holcomb, P. J., Ditman, T., & Kuperberg, G. R. (2010). Electrophysiological insights into the processing of nominal metaphors. Neuropsychologia, 48(7), 1965–1984. https://doi.org/10.1016/j.neuropsychologia.2010.03.017
  • Delorme, A., & Makeig, S. (2004). EEGLAB: An open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. Journal of Neuroscience Methods, 134(1), 9–21. https://doi.org/10.1016/j.jneumeth.2003.10.009
  • Di Nuovo, S., Guarnera, M., & Castellano, S. (2014). Mental imagery test. Hogrefe.
  • Ervas, F. (2019). Visual metaphor, imagined communities and deferred rationality. Sistemi Intelligenti, 31(3), 413–438. https://doi.org/10.1422/95083
  • Ervas, F. (2021). Metaphor, ignorance and the sentiment of (ir)rationality. Synthese, 198(7), 6789–6813. https://doi.org/10.1007/s11229-019-02489-y
  • Fenko, A., de Vries, R., & van Rompay, T. (2018). How strong is your coffee? The influence of visual metaphors and textual claims on consumers’ flavor perception and product evaluation. Frontiers in Psychology, 9, 53. https://doi.org/10.3389/fpsyg.2018.00053
  • Forceville, C. J. (2008). Metaphor in pictures and multimodal representations. In R. W. Gibbs (Ed.), The Cambridge handbook of metaphor and thought (pp. 462–482). Cambridge University Press. https://doi.org/10.1017/CBO9780511816802.028
  • Forceville, C. J. (2016). Pictorial and multimodal metaphor. In N. M. Klug & H. Stöckl (Eds.), Handbuch Sprache im multimodalen Kontext (pp. 241–260). De Gruyter. https://doi.org/10.1515/9783110296099-011
  • Forceville, C. J., & Urios-Aparisi, E. (Eds.). (2009). Multimodal metaphor. Mouton de Gruyter. https://doi.org/10.1515/9783110215366
  • Ganis, G., Kutas, M., & Sereno, M. I. (1996). The search for “Common sense”: An electrophysiological study of the comprehension of words and pictures in reading. Journal of Cognitive Neuroscience, 8(2), 89–106. https://doi.org/10.1162/jocn.1996.8.2.89
  • Gibbs, R. W. (1994). The poetics of mind: Figurative thought, language, and understanding. Cambridge University Press.
  • Gibbs, R. W., & Colston, H. L. (2012). Interpreting figurative meaning. Cambridge University Press. https://doi.org/10.1017/cbo9781139168779
  • Hirschfeld, G., Feldker, K., & Zwitserlood, P. (2012). Listening to “Flying ducks”: Individual differences in sentence-picture verification investigated with ERPs. Psychophysiology, 49(3), 312–321. https://doi.org/10.1111/j.1469-8986.2011.01315.x
  • Hornof, A. J. (2004). Cognitive strategies for the visual search of hierarchical computer displays. Human-Computer Interaction, 19(3), 183–223. https://doi.org/10.1207/S15327051HCI1903_1
  • Ifantidou, E. (2021). Non-propositional effects in verbal communication: The case of metaphor. Journal of Pragmatics, 181, 6–16. https://doi.org/10.1016/j.pragma.2021.05.009
  • Jung, T.-P., Makeig, S., Humphries, C., Lee, T.-W., McKeown, M. J., Iragui, V., & Sejnowski, T. J. (2000). Removing electroencephalographic artifacts by blind source separation. Psychophysiology, 37(2), 163–178. https://doi.org/10.1111/1469-8986.3720163
  • Katz, A. N., Paivio, A., Marschark, M., & Clark, J. M. (1988). Norms for 204 literary and 260 nonliterary metaphors on 10 psychological dimensions. Metaphor and Symbolic Activity, 3(4), 191–214. https://doi.org/10.1207/s15327868ms0304_1
  • Ketelaar, P. E., Gisbergen, M. S. V., & Beentjes, J. (2012). Interpretation of highly visual ‘Open’ advertisements in Dutch magazines. Journal of Visual Literacy, 31(1), 23–52. https://doi.org/10.1080/23796529.2012.11674693
  • Knauff, M., & Johnson-Laird, P. N. (2002). Visual imagery can impede reasoning. Memory & Cognition, 30(3), 363–371. https://doi.org/10.3758/BF03194937
  • Kutas, M., & Federmeier, K. D. (2011). Thirty years and counting: Finding meaning in the N400 component of the event-related brain potential (ERP). Annual Review of Psychology, 62(1), 621–647. https://doi.org/10.1146/annurev.psych.093008.131123
  • Lagerwerf, L., & Meijers, A. (2008). Openness in metaphorical and straightforward advertisements: Appreciation effects. Journal of Advertising, 37(2), 19–30. https://doi.org/10.2753/JOA0091-3367370202
  • Lai, V. T., & Curran, T. (2013). ERP evidence for conceptual mappings and comparison processes during the comprehension of conventional and novel metaphors. Brain and Language, 127(3), 484–496. https://doi.org/10.1016/j.bandl.2013.09.010
  • Lai, V. T., Curran, T., & Menn, L. (2009). Comprehending conventional and novel metaphors : An ERP study. Brain Research, 1284, 145–155. https://doi.org/10.1016/j.brainres.2009.05.088
  • Lau, E. F., Phillips, C., & Poeppel, D. (2008). A cortical network for semantics: (de)constructing the N400. Nature Reviews. Neuroscience, 9(12), 920–933. https://doi.org/10.1038/nrn2532
  • Ma, Q., Hu, L., Xiao, C., Bian, J., Jin, J., & Wang, Q. (2016). Neural correlates of multimodal metaphor comprehension: Evidence from event-related potentials and time-frequency decompositions. International Journal of Psychophysiology, 109, 81–91. https://doi.org/10.1016/j.ijpsycho.2016.09.007
  • Malhi, S. K., & Buchanan, L. (2018). A test of the symbol interdependency hypothesis with both concrete and abstract stimuli. PLoS ONE, 13(3), e0192719. https://doi.org/10.1371/journal.pone.0192719
  • Marelli, M. (2017). Word-embeddings Italian semantic spaces: A semantic model for psycholinguistic research. Psihologija, 50(4), 503–520. https://doi.org/10.2298/PSI161208011M
  • Marks, D. F. (1973). Visual imagery differences in the recall of pictures. British Journal of Psychology, 64(1), 17–24. https://doi.org/10.1111/J.2044-8295.1973.TB01322.X
  • Marschark, M., & Hunt, R. R. (1985). On memory for metaphor. Memory & Cognition, 13(5), 413–424. https://doi.org/10.3758/BF03198454
  • McPherson, W. B., & Holcomb, P. J. (1999). An electrophysiological investigation of semantic priming with pictures of real objects. Psychophysiology, 36(1), 53–65. https://doi.org/10.1017/S0048577299971196
  • McQuarrie, E. F., & Mick, D. G. (1996). Figures of rhetoric in advertising language. Journal of Consumer Research, 22(4), 424. https://doi.org/10.1086/209459
  • McQuarrie, E. F., & Mick, D. G. (2003). The contribution of semiotic and rhetorical perspectives to the explanation of visual persuasion in advertising. In L. M. Scott & R. Batra (Eds.), Persuasive imagery: A consumer response perspective (pp. 191–222). Lawrence Erlbaum Associates.
  • McQuarrie, E. F., & Phillips, B. J. (2005). Indirect persuasion in advertising: How consumers process metaphors presented in pictures and words. Journal of Advertising, 34(2), 7–20. https://doi.org/10.1080/00913367.2005.10639188
  • Mintzer, M. Z., & Snodgrass, J. G. (1999). The picture superiority effect: Support for the distinctiveness model. The American Journal of Psychology, 112(1), 113–146. https://doi.org/10.2307/1423627
  • Mon, S. K., Nencheva, M., Citron, F. M. M., Lew-Williams, C., & Goldberg, A. E. (2021). Conventional metaphors elicit greater real-time engagement than literal paraphrases or concrete sentences. Journal of Memory and Language, 121, 104285. https://doi.org/10.1016/j.jml.2021.104285
  • Muraki, E. J., & Pexman, P. M. (2021). Simulating semantics: Are individual differences in motor imagery related to sensorimotor effects in language processing? Journal of Experimental Psychology: Learning Memory and Cognition, 47(12), 1939–1957.
  • Noveck, I. A., Bianco, M., & Castry, A. (2001). The costs and benefits of metaphor. Metaphor and Symbol, 16(1–2), 109–121. https://doi.org/10.1080/10926488.2001.9678889
  • Ojha, A., Ervas, F., Gola, E., & Indurkhya, B. (2019). Similarities and differences between verbal and visual metaphor processing: An EEG study. Multimodal Communication, 8(2), 1–13. https://doi.org/10.1515/mc-2019-0006
  • Ojha, A., Indurkhya, B. (2016). On the role of perceptual features in metaphor comprehension. In E. Gola & F. Ervas (Eds.), Metaphor and communication (pp. 147–170). John Benjamins.
  • Oldfield, R. C. (1971). The assessment and analysis of handedness: The Edinburgh inventory. Neuropsychologia, 9(1), 97–113. https://doi.org/10.1016/0028-3932(71)90067-4
  • Oostenveld, R., Fries, P., Maris, E., & Schoffelen, J. M. (2011). FieldTrip: Open source software for advanced analysis of MEG, EEG, and invasive electrophysiological data. Computational Intelligence and Neuroscience, 2011, 156869. https://doi.org/10.1155/2011/156869
  • Ortiz, M. J., Grima Murcia, M. D., & Fernandez, E. (2017). Brain processing of visual metaphors: An electrophysiological study. Brain and Cognition, 113, 117–124. https://doi.org/10.1016/j.bandc.2017.01.005
  • O’Sullivan, M., Lepage, M., Bouras, M., Montreuil, T., & Brodeur, M. B. (2012). North-American norms for name disagreement: Pictorial stimuli naming discrepancies. PLoS ONE, 7(10), e47802. https://doi.org/10.1371/journal.pone.0047802
  • Özyürek, A., Willems, R. M., Kita, S., & Hagoort, P. (2007). On-line integration of semantic information from speech and gesture: Insights from event-related brain potentials. Journal of Cognitive Neuroscience, 19(4), 605–616. https://doi.org/10.1162/jocn.2007.19.4.605
  • Paivio, A. (1971). Imagery and language. In S. J. Segal (Ed.), Imagery (pp. 7–32). Elsevier. https://doi.org/10.1016/B978-0-12-635450-8.50008-X
  • Pavan, A., & Baggio, G. (2013). Linguistic representations of motion do not depend on the visual motion system. Psychological Science, 24(2), 181–188. https://doi.org/10.1177/0956797612450882
  • Pérez-Sobrino, P. (2016). Multimodal metaphor and metonymy in advertising: A corpus-based account. Metaphor and Symbol, 31(2), 73–90. https://doi.org/10.1080/10926488.2016.1150759
  • Peterson, M. O. (2019). Aspects of visual metaphor: An operational typology of visual rhetoric for research in advertising. International Journal of Advertising, 38(1), 67–96. https://doi.org/10.1080/02650487.2018.1447760
  • Peterson, M. O., Wise, K., Ren, Y., Wang, Z., & Yao, J. (2017). Memorable metaphor: How different elements of visual rhetoric affect resource allocation and memory for advertisements. Journal of Current Issues and Research in Advertising, 38(1), 65–74. https://doi.org/10.1080/10641734.2016.1233155
  • Pinheiro, J. C., & Bates, D. M. (2000). Mixed-effects models in S and S-PLUS. Springer-Verlag.
  • Pynte, J., Besson, M., Robichon, F. H., & Poli, J. (1996). The time-course of metaphor comprehension: An event-related potential study. Brain and Language, 55(3), 293–316. https://doi.org/10.1006/brln.1996.0107
  • Rataj, K. (2014). Surfing the brainwaves of metaphor comprehension. Poznan Studies in Contemporary Linguistics, 50(1), 55–73. https://doi.org/10.1515/psicl-2014-0004
  • Rayner, K. (1998). Eye movements in reading and information processing: 20 years of research. Psychological Bulletin, 124(3), 372–422. https://doi.org/10.1037/0033-2909.124.3.372
  • R Core Team. (2018). R: A Language and Environment for Statistical Computing. Vienna: R Foundation for Statistical Computing.
  • Reid, J. N., Al-Azary, H., & Katz, A. N. (2020). Metaphors: Where the neighborhood in which one resides interacts with (interpretive) diversity. Proceedings of the 42nd annual meeting of the cognitive science society (pp. 551–556).
  • Roncero, C., & de Almeida, R. G. (2015). Semantic properties, aptness, familiarity, conventionality, and interpretive diversity scores for 84 metaphors and similes. Behavior Research Methods, 47(3), 800–812. https://doi.org/10.3758/s13428-014-0502-y
  • Schendan, H. E., & Ganis, G. (2012). Electrophysiological potentials reveal cortical mechanisms for mental imagery, mental simulation, and grounded (embodied) cognition. Frontiers in Psychology, 3, 329. https://doi.org/10.3389/fpsyg.2012.00329
  • Schumacher, P. B. (2014). Content and context in incremental processing: “The ham sandwich” revisited. Philosophical Studies, 168(1), 151–165. https://doi.org/10.1007/s11098-013-0179-6
  • Searle, J. (1979). Metaphor, and A. Ortony (Ed.), Metaphor and thought (pp. 83–111). Cambridge University Press. https://doi.org/10.1017/CBO9781139173865
  • Shen, Z. Y., Tsai, Y. T., & Lee, C. L. (2015). Joint influence of metaphor familiarity and mental imagery ability on action metaphor comprehension: An event-related potential study. Language and Linguistics, 16(4), 615–637. https://doi.org/10.1177/1606822X15583241
  • Sitnikova, T., Holcomb, P. J., Kiyonaga, K. A., & Kuperberg, G. R. (2008). Two neurocognitive mechanisms of semantic integration during the comprehension of visual real-world events. Journal of Cognitive Neuroscience, 20(11), 2037–2057. https://doi.org/10.1162/jocn.2008.20143
  • Sperber, D., & Wilson, D. (2008). A deflationary account of metaphors. In R. W. Gibbs (Ed.), The Cambridge handbook of metaphor and thought (pp. 84–105). Cambridge University Press. https://doi.org/10.1017/CBO9781139028370.007
  • Spotorno, N., Cheylus, A., Van Der Henst, J. B., & Noveck, I. A. (2013). What’s behind a P600? Integration operations during irony processing. PLoS ONE, 8(6), e66839. https://doi.org/10.1371/journal.pone.0066839
  • Thibodeau, P. H., Matlock, T., & Flusberg, S. J. (2019). The role of metaphor in communication and thought. Linguistics and Language Compass, (13), e12327. July 2018. https://doi.org/10.1111/lnc3.12327
  • Tremblay, A., & Newman, A. J. (2015). Modeling nonlinear relationships in ERP data using mixed-effects regression with R examples. Psychophysiology, 52(1), 124–139. https://doi.org/10.1111/PSYP.12299
  • Utsumi, A. (2005). The role of feature emergence in metaphor appreciation. Metaphor and Symbol, 20(3), 151–172. https://doi.org/10.1207/s15327868ms2003_1
  • Vance, K., & Virtue, S. (2011). Metaphoric advertisement comprehension: The role of the cerebral hemispheres. Journal of Consumer Behaviour, 10(1), 41–50. https://doi.org/10.1002/cb.345
  • Van Mulken, M., Van Hooft, A., & Nederstigt, U. (2014). Finding the tipping point: Visual metaphor and conceptual complexity in advertising. Journal of Advertising, 43(4), 333–343. https://doi.org/10.1080/00913367.2014.920283
  • Weiland, H., Bambini, V., & Schumacher, P. B. (2014). The role of literal meaning in figurative language comprehension: Evidence from masked priming ERP. Frontiers in Human Neuroscience, 8, 583. https://doi.org/10.3389/fnhum.2014.00583
  • West, W. C., & Holcomb, P. J. (2000). Imaginal, semantic, and surface-level processing of concrete and abstract words: An electrophysiological investigation. Journal of Cognitive Neuroscience, 12(6), 1024–1037. https://doi.org/10.1162/08989290051137558
  • Yus, F. (2009). Visual metaphor versus verbal metaphor: A unified account. In C. J. Forceville & E. Urios-Aparisi (Eds.), Multimodal metaphor (pp. 147–172). Mouton de Gruyter. https://doi.org/10.1515/9783110215366.3.147
  • Zwaan, R. A., & Pecher, D. (2012). Revisiting mental simulation in language comprehension: Six replication attempts. PLoS ONE, 7(12), e51382. https://doi.org/10.1371/journal.pone.0051382

Reprints and Corporate Permissions

Please note: Selecting permissions does not provide access to the full text of the article, please see our help page How do I view content?

To request a reprint or corporate permissions for this article, please click on the relevant link below:

Academic Permissions

Please note: Selecting permissions does not provide access to the full text of the article, please see our help page How do I view content?

Obtain permissions instantly via Rightslink by clicking on the button below:

If you are unable to obtain permissions via Rightslink, please complete and submit this Permissions form. For more information, please visit our Permissions help page.