How we remember what we can do

Abstract

According to the motor simulation theory, the knowledge we possess of what we can do is based on simulation mechanisms triggered by an off-line activation of the brain areas involved in motor control. Action capabilities memory does not work by storing some content, but consists in the capacity, rooted in sensory-motor systems, to reenact off-line action sequences exhibiting the range of our powers. In this paper, I present several arguments from cognitive neuropsychology, but also first-person analysis of experience, against this hypothesis. The claim that perceptual access to affordances is mediated by motor simulation processes rests on a misunderstanding of what affordances are, and comes up against a computational reality principle. Motor simulation cannot provide access to affordances because (i) the affordances we are aware of at each moment are too many for their realization to be simulated by the brain and (ii) affordances are not equivalent to currently or personally feasible actions. The explanatory significance of the simulation theory must then be revised downwards compared to what is claimed by most of its advocates. One additional challenge is to determine the prerequisite, in terms of cognitive processing, for the motor simulation mechanisms to work. To overcome the limitations of the simulation theory, I propose a new approach: the direct content specification hypothesis. This hypothesis states that, at least for the most basic actions of our behavioral repertoire, the action possibilities we are aware of through perception are directly specified by perceptual variables characterizing the content of our experience. The cognitive system responsible for the perception of action possibilities is consequently far more direct, in terms of cognitive processing, than what is stated by the simulation theory. To support this hypothesis I review evidence from current neuropsychological research, in particular data suggesting a phenomenon of ‘fossilization’ of affordances. Fossilization can be defined as a gap between the capacities that are treated as available by the cognitive system and the capacities this system really has at its disposal. These considerations do not mean that motor simulation cannot contribute to explain how we gain perceptual knowledge of what we can do based on the memory of our past performances. However, when precisely motor simulation plays a role and what it is for exactly currently remain largely unknown.

• cognitive neuropsychology
• phenomenology
• action memory
• motor simulation
• mental imagery
• affordance
• action possibility Responsible Editor: Mathieu Hainselin, Université de Picardie Jules Verne, France.

Acknowledgements

I am grateful to Andrew D. Wilson and the anonymous reviewers of the Socioaffective Neuroscience & Psychology Journal for their insightful comments on earlier versions of this paper.

Notes

¹The simulation model is used to account for many other cognitive abilities, especially the ability to attribute mental states to others (mind reading) or, more radically, to understand others as intentional agents. These works will not be discussed here.

²As Witt and Proffitt (2008) explain, ‘the outcome of the simulation, rather than the perceiver's ability at the time of estimating distance, influences perception’.

³I have presented some of these arguments in another article (Declerck, 2013). To avoid building from scratch, I partly draw on them in the present paper.

⁴A negative support to this claim can be found in the spatial disorientation phenomenon. When you are disoriented – you are somewhere but you are incapable of determining where you are, e.g. in the immediate period following an accident – what happens is precisely that this field of possibilities is erased from your awareness of the situation.

⁵Based on Heidegger's phenomenological analysis of our everyday coping with objects, Turner (2005) makes a similar claim: affordances participate in a network and we never perceive them in isolation.

⁶A similar argument was used by O'Regan and Noë (2001) against internalist approaches to perception.

⁷An additional point which further undermines the computational credibility of ST is related to what can be called negative affordances. One can perceive objects as – more or less immediately – reachable, but obviously one can also perceive them as out of reach, i.e. as not reachable given one's current respective position and action resources. If, as claimed by ST advocates, motor simulation mechanisms explain the calibration of visual distances on the reaching or moving-close-to capabilities of the perceiving agent, does it mean that to register a visual object as out-of-reach one must run a simulation of the reaching action and this simulation must end with a negative output? That is to say, is the perception of negative affordances the outcome of a failing simulation? That seems highly improbable.

⁸It is worth noting, however, that these data deal with the capacity to perform a given action while engaging into the explicit (i.e. motor imagery) or implicit simulation of another action, and that, currently, there seems to be no data demonstrating that the brain cannot manage several actions simultaneously when processing them only covertly and in a purely non-conscious way. It is a widespread observation that in many cognitive tasks performances are better when the task is achieved without (or with low) conscious monitoring (see e.g. Rousselet, Fabre-Thorpe, & Thorpe, 2002, for the demonstration that subjects are able to process in parallel several images presented simultaneously so as to extract information about high-level (i.e. semantic) objects properties). One could thus expect better performances for purely implicit motor simulation, than for motor imagery or overt action execution, or a combination of both. This issue needs to be investigated, but, whatever will be found, it will not destroy my argument. For even if it turns out that the motor system is able to process simultaneously more actions than we suppose, it is highly unlikely that it can manage the (huge) number of action possibilities we are at each instant aware of –though implicitly and peripherally (see page 6). The claim that each affordance we are aware of results from a corresponding ongoing motor simulation process remains in any case implausible. I am grateful to an anonymous referee for drawing my attention to this issue and empirical data.

⁹More precisely: the action of ‘picking-up’ the glass will be anticipated as realizable. As I will explain below, perceiving that an object offers an affordance such as the pick-up-ability must not be confused with perceiving (or anticipating) that the action is realizable (i.e. actualizable in the current circumstances). The glass remains ‘pick-up-able’ even if I cannot pick it up right now and it remains ‘pick-up-able’ even if I cannot pick it up (maybe others can).

¹⁰Note that the overcapacity argument presented on page 6 is directed toward the claim made by ST (at least by some of its advocates) that one's perceptual access to affordances is always subtended by motor simulation processes, whether or not the object falls within one's peripersonal space. To that extent, these empirical data do not weaken the argument.

¹¹The claim that affordances are not enslaved to immediate actualizability probably does not apply equally well to all kinds of so-called affordances. For instance, it could be stated that reachability necessarily refers to the possibility of reaching the objects here and now, given the relative position of our body and the reach of our arm. Current exploitability would be in that case a constitutive component of the affordance. But this is clearly not the case for other types of affordances: a given solid surface affords support for standing and walking even if I am not in position to use it or – more radically – even if I am, for any reason, unable to use it (I might be paralyzed). That's one reason why the distinctions proposed here are important, whereas the notion of affordance, as it is generally used, is probably too general to be relevant. The notion of reachability only applies to the action possibilities of types (ii) and (iii). No real object is in principle reachable (or all objects are). Solid visible objects are reachable when certain proximity conditions are fulfilled. It means that if we accept that the notion of affordance only applies to action possibilities of type (i), reachability is not an affordance.

¹²For instance, Garbarini and Adenzato (2004) explain: ‘only by virtually executing the action can we understand the relational significance of the object, i.e. the affordance it offers’.

¹³This issue recalls a well-known objection to the frame approach in artificial intelligence: if the knowledge the machine has of its situation amounts to a set of descriptive propositions (declarative knowledge stored in a base of facts), the problem arises of how to determine which propositions must be reevaluated when a change occurs in the environment or in the situation of the machine, e.g. its position. As Fodor explains: ‘How […] does the machine's program determine which beliefs the robot ought to reevaluate given that it has embarked upon some or other course of action?’ (Fodor, 1983, p. 114).

¹⁴Exactly the same objection can be made against the following claim of Gallese (2000): ‘To observe objects is therefore equivalent to automatically evoking the most suitable motor program required to interact with them. Looking at objects means to unconsciously “simulate” a potential action. In other words, the object representation is transiently integrated with the action-simulation (the ongoing simulation of the potential action)’. How does the cognitive system know in advance what are the ‘most suitable motor programs required to interact with [the objects]’?

¹⁵See especially Reed (1988, p. 231) who quotes a handwritten note from Gibson where this idea is expressed with particular clarity.

¹⁶‘Although he never discusses the issues in quite these terms, it is reasonably evident from Gibson's practice that he wishes to distinguish between what is picked up and what is directly perceived. In fact, Gibson ultimately accepts something like our first constraint – that what is picked up in visual perception is only certain properties of the ambient light array. Gibson is thus faced with the problem of how, if not by inferential mediation, the pickup of such properties of light could lead to perceptual knowledge of properties of the environment. That is: how, if not by inference, do you get from what you pick up about the light to what you perceive about the environmental object that the light is coming from?’ (Fodor & Pylyshyn, 1981, p. 143).

¹⁷This characterization of sensational content, because of its purely negative character, is not very satisfying: it tells what sensational content is not, but it does not tell what sensational content is. A more acceptable definition would build on Husserl's characterization of experience as organized through layers. What is sensational for this approach can be specified based on a series of ‘reductions’, i.e. abstraction operations. See e.g. Husserl (1907, §. 14 and §. 15).

¹⁸For sake of simplicity, we assume that our knowledge of what can be done in the environment has a belief format (see page 8). More exactly, this knowledge is the object of dispositional beliefs which are automatically caused by perception: when perceiving an object O, we know what we can do with (or given) O insofar as we possess the belief that actions {A₁,…, A_n} are possible with O.

¹⁹Note that this formulation of the direct content specification hypothesis does not entail a commitment to representationalism, in the classical sense of assuming the existence of neural states that stand in for state of affairs in the external world (see e.g. Clark & Toribio, 1994; Degenaar & Myin, 2014). Most accounts of perception assuming a form of representationalism make use of the concept of content and most anti-representationalist accounts build on an attack on this concept (see e.g. Hutto & Myin, 2013). However, the concept of content, and especially of representational content, which is used here is a purely phenomenological concept: it is not stated that neural states (or any part of the organism of the perceiving agent) represent properties of the external world; it is stated that perceptual awareness, understood as an intentional state, is something that presents the world to be in this or that way. Or, to say it a bit differently, in order to describe the perceptual experience of a subject, you need to take into account that this experience presents the world as being this or that way.

²⁰Whether and to what extent Gibson's direct theory of perception can (also) be considered as a first-person theory of perception might be discussed. My position is that Gibson's theory is better interpreted as a purely third-person account of perception. Undoubtedly, many of Gibson's claims seem to have some phenomenological significance. However, these descriptions are not elaborated in a systematic and methodologically sound way: Gibson does not rely on a specific descriptive method or make use of a dedicated conceptual system; to a large extent, the ‘first-person’ descriptions he proposes remain ‘naïve’, in this sense. In addition, perception is not analyzed and studied by Gibson as a conscious state with phenomenological properties, but as a behavioral category: some behavioral patterns are interpreted as patterns of ‘perceptual activity’. As Turvey (1974, p. 166) explains, for Gibson, ‘perception of the environment corresponds simply and solely to detection of […] variables of stimulation’. Consequently, Gibson's theory simply does not need to include a first-person account of perceptual experience to be a theory of perception. To explain how perception is possible, it is sufficient to explain how the detection of these variables can occur.

²¹The principle of the fossilization phenomenon has been presented in Declerck and Gapenne (2009) and Declerck (2012) on the basis of neuropsychological and phenomenological data.

²²The observations of Iriki et al. have been replicated in humans by studies involving line bisection tasks with healthy subjects (Longo & Lourenco, 2006, 2007) and subjects suffering from neglect (Berti & Frassinetti, 2000; Cowey, Small, & Ellis, 1994; Halligan & Marshall, 1991; Pegna et al., 2001), distance estimation tasks (Witt & Proffitt, 2008; Witt et al., 2005), intermodal extinction (Farnè, Iriki, & Làdavas, 2005; Farnè & Ládavas, 2000; Maravita, Husain, Clarke, and Driver, 2001), and intermodal interference (Maravita, Spence, & Driver, 2003; Maravita, Spence, Kennett, & Driver, 2002; Maravita, Spence, Sergent, & Driver, 2002). Several studies also corroborate the observation made by Iriki et al. (1996) that the active use of the tool is necessary for the phenomenon of reconfiguration of body schema and peripersonal space to occur (see e.g. Farnè et al., 2005; Farnè & Ládavas, 2000).