Words speak louder than pictures for action concepts: an eyetracking investigation of the picture superiority effect in semantic categorisation

ABSTRACT

An evolving consensus holds that actions and objects are supported by at least partially distinct neurocognitive substrates. In addition, representational modality (e.g. pictures vs. words) moderates the speed and accuracy of semantic access for both word classes. The picture superiority effect refers to the advantage that pictures manifest relative to words across a range of object recognition and memory recall tasks. While the effect has been investigated exclusively in the context of static objects, it is unclear whether this modality advantage extends to the processing of actions and/or verbs. We report two eyetracking experiments examining the dynamics of modality-specific access to actions via associative semantic judgement tasks for actions depicted as pictures or words. A range of eyetracking and behavioural measures revealed that participants consistently showed a reversal of the picture superiority effect for action semantic categorisation. These results challenge the privileged access hypothesis and demonstrate further distinctiveness of the action-object processing dichotomy. We discuss implications for action semantic processing.

KEYWORDS:

• Semantic memory
• verb processing
• eyetracking
• semantic access
• action concept

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes

1. The Hub-and-Spoke model of semantic cognition is another unitary model premised upon amodal semantic convergence within regions of anterior temporal cortex (Patterson, Nestor, & Rogers, 2007). Patients with progressive atrophy of these regions have been reported to experience semantic dementia, now more commonly known as semantic variant primary progressive aphasia (svPPA) (Gorno-Tempini et al., 2011; Hodges, Patterson, Oxbury, & Funnell, 1992). Patients with svPPA typically show homogeneous impairment for actions and objects, and such impairments tend to transcend representational modality (Corbett, Jefferies, Ehsan, & Lambon Ralph, 2009; Lambon Ralph, Sage, Jones, & Mayberry, 2010; Pulvermuller et al., 2010). The relative stability of this impairment across categories and modalities lends support to the presence of a single amodal semantic store (Binney, Embleton, Jefferies, Parker, & Lambon Ralph, 2010).

2. It should be noted that the neural substrates of action concepts are represented not only in the motor regions but also in the lateral temporal region (see Kemmerer, 2015 for discussion). The underlying mechanism of these representations is still debatable. While a strong embodied approach emphasizes the automaticity and necessity of motor regions during action processing (Hauk, Johnsrude, & Pulvermuller, 2004; Pulvermuller, Harle, & Hummel, 2001), opponents argue that the lateral temporal region is also engaged, and thus activation in the motor regions alone is not sufficient for the conceptual representation of actions (Bedny et al., 2008; Kemmerer, Rudrauf, Manzel, & Tranel, 2012; Papeo et al., 2015; Peelen, Romagno, & Caramazza, 2012; Watson, Cardillo, Ianni, & Chatterjee, 2013). Convergent evidence would be essential to determine how and to what extent actions are encoded in the brain.

Additional information

Funding

This work was supported by US Public Health Service grant DC013063.