Interface Theory vs Gibson: An Ontological Defense of the Ecological Approach

ABSTRACT

Interface theory is the hypothesis that inferential, representational theories of perception entail that fitness, not truth, dictates the evolution of perceptual systems. They show, with simulations, that “veridical” perceptual mappings (ones that preserve at least some of the structure of the world) are routinely out-competed by “non-veridical” interfaces (ones that make no attempt to preserve that structure). They therefore take particular aim at the direct perception, ecological approach to perception and work to show that such a system, even if technically an option, would never be selected for by evolution. This paper defends the ecological approach from this novel, existential attack by showing that the ecological hypothesis is so different in kind to the inferential, representational view of perception that it simply falls outside the scope of interface theory’s critiques; ecological psychology remains a viable scientific endeavor. This analysis will show that, far from being a radical new approach to perception, interface theory is simply a clear and elegant formalization of mainstream representational psychology, and any implications interface theory may have belong solely to that branch of science.

KEYWORDS:

• Interface theory
• ecological psychology
• perception
• representation
• ontology

Introduction

Hoffman et al. (2015; hence HS&P) propose a supposedly radical take on the nature of perceptual experience in the form of their interface theory (see also Hoffman, 2008, 2009, 2011; Hoffman & Singh, 2012; Mark et al., 2010). They begin with the Establishment¹ assumption that perception is an inferential, representational, mental process. This assumption is a consequence of the hypothesis that the objects of perception are observer-independent properties (the kind of properties measured by physics, e.g., mass, spatial extent). To support behavior, these properties must be transformed into behaviorally relevant, observer-dependent properties (e.g., is that a mass I can lift? Is that a gap I can cross?). For HS&P and other representational researchers, this transformation is implemented by a computational mental representation which infers the behaviorally relevant property from the presence of the observer-independent properties. HS&P formalize this to say that these mental representations implement a mapping function to convert the physical world into a behaviorally relevant perceptual experience. They then call these functions interfaces, and note that these mapping functions can, in principle, vary in the degree to which they preserve the structure of the physical world. In a series of evolutionary simulations, HS&P claim to compete all possible types of mapping functions against one another, and formally demonstrate the surprising result that veridical interfaces (mappings that preserve at least some of the structure of the environment) are routinely out-competed into extinction by non-veridical interfaces (mappings built in whatever form required to get to the behaviorally-relevant experience and, more specifically, to optimize fitness). Perception, they argue, must therefore be a non-veridical but adaptive interface that hides the messy details of the “true” world and instead presents us with a series of behaviorally useful fictions, fictions they say include things like “objects” and “events”.

One theory of perception that seems to immediately lose in this scenario is James J Gibson’s direct perception, ecological approach (Gibson, 1966, 1979; Michaels & Carello, 1981; Turvey et al., 1981). Gibson rejected the need for internal mental representations performing information processing. Instead, he proposed a theory of ecological information which explains how behaviorally relevant, observer-dependent properties can be the primary objects of perception. If perception does not require inference to connect us to the behaviorally relevant world, then perception should be veridical, at least to some extent. If, however, HS&P are right and evolution will always favor non-veridical perceptual systems over any kind of veridical ones, then evolution will never produce a Gibsonian organism, and the ecological approach is therefore dead on arrival. HS&P of course argue precisely this.

This is a novel challenge to the ecological approach; most critiques focus on the limits of what information can specify and therefore what behaviors it can or cannot support (e.g., catching a fly ball vs imagining catching a fly ball; see Clark & Toribio, 1994). Interface theory instead claims that evolution will never produce an ecological perceptual system of any kind; that only a representational system can support perception. This paper defends Gibson and the ecological approach from the formalized objections HS&P raise, in order to demonstrate that direct perception remains a viable competitor to the representational approach.

The paper will work as follows. I will briefly review interface theory. I will then detail the new interface theory objection, that Gibson’s ecological approach is a form of veridical perceptual mapping strategy that falls victim to the various results from the evolutionary simulations. My primary line of defense is philosophical,² and it is the claim that the ecological and inferential approaches to perception are so ontologically different in kind that the challenge posed by HS&P comes nowhere near Gibson. I will detail the two ontologies and their implications, then detail how the modern ecological approach has formalized its ontology into a science of affordances and perceptual information. This analysis will show three things. First, interface theory is not so radical; it is actually a fairly standard inferential, representational theory of perception. More importantly, the various radical results actually emerge from the long-standing and unsolved problem of representational grounding (e.g., Harnad, 1990; Searle, 1980), which interface theory embraces, rather than solves. Second, it will make it clear that the radically different understanding of perception is actually the ecological approach, and that this difference runs deep enough that the results of interface theory have no implications for Gibson’s theory. Third, it will show that only the ecological ontology can resolve the grounding problem, which really needs solving (Golonka & Wilson, 2019a).

Interface Theory

What is the relationship between the structure of the world and the structure of our perceptual experience of that world? Is our experience veridical, or an adaptive fiction? To answer this, HS&P claim to identify and formally describe the full range of possible perceptual mapping strategies in terms of set theory (pg.1482), and then place the strategies in simulated evolutionary competition with one another to see which prevails.

Their analysis proceeds as follows. There is perceptual experience, described as a set of states X, some of which are behaviorally relevant events. There is also a world, described as a set of states W, some of which are behaviorally relevant events. Different perceptual mapping strategies are then expressed as different functions P that relate world states W to perceptual states X (P:W→X). HS&P then describe a continuum of functions P that vary in how veridically they map W→X by relaxing the degree to which P preserves W structure into X. These strategies³ range from omniscient realism (where perceptual experience and the world are identical sets and therefore world structure is veridically preserved in perceptual experience) through naïve realism (where perceptual experience is a subset of world states and that subset veridically preserves the structure of the world) to critical realism (where perceptual experience is not a subset of world states but does veridically preserve at least some of the structure of the world set anyway). The final strategy is the interface, in which perceptual experience is not a subset of world states and there is no requirement for perception to veridically preserve any of the structure in the world set.

HS&P then ran evolutionary simulations (pg.1485) in which agents implementing the various strategies compete with each other for resources in a simple world. The agents’ ability to reproduce is evaluated using a fitness function, and the winner identified using a replicator equation. HS&P describe one example of a world set that includes one resource, the amount of which can range from 0 to 100, and agents with perceptual experience sets that consists of four possible states. The resource has a Gaussian pay-off function for the agents; too little or too much is bad, with amounts centered on 50 the best. The question is, which way of mapping world states into perceptual states produces the most adaptive behavior (best tracks the pay-off function)?

All realist strategies must organize their perceptual states so as to replicate at least a subset of the structure in the world. In this example, the world structure is resource amount. One state could be used to represent 0–25 of the resource, another 25–50, then 50–75 and finally 75–100. The problem here is that this is not an effective way of representing the Gaussian pay-off function. Categorizing amounts from 50 to 75 as “the same” in perceptual experience means lumping in many non-optimal amounts with optimal ones, and the resulting behavior will not be optimally adaptive.

Interface strategies are not limited this way, and so perceptual experience is allowed to be organized in a way that has nothing to do with world structure like resource quantity if that’s not useful. An interface strategy can use states to categorize not by resource amount, but by payoff amount (e.g., use one state to represent “low payoff,” which happens close to both 0 and 100 amount of resource).

Whenever payoff is a non-monotonic function of resource quantity (which HS&P claim is typical) then realist strategies are unable to represent the payoff function effectively and thus produce less adaptive behavior. Interfaces can represent the payoff function, however, and therefore HS&P find that “… strict interface strategies that are tuned to fitness routinely drive naïve realist and critical realist strategies to extinction” (page 1486). In order to support adaptive behavior, they argue that real perception should therefore implement an interface perceptual mapping strategy.

This has a big implication. If perception is an interface, then the elements of our perceptual experience (objects, events, etc.) might simply be fictions that do not exist in the world. Instead, they would be cognitive inventions that efficiently encode the payoff function that describes how to adaptively interact with the world. HS&P make the analogy to icons in a windows-based computer interface (pg.1488). There is no literal “file on a desktop” in my computer, but this representation of the internal structure of my computer’s memory is an effective way for me to interact quickly and accurately with that actual internal structure. In the same way, HS&P suggest that there may be nothing sensibly described as “objects” out in the world, and that these are merely icons in our perceptual experience that allow us to interact adaptively with the world:

Formally, the interface theory of perception proposes that the perceptual strategies of H. sapiens and, indeed, of all organisms are, generically, strict interface strategies. Recall that this means, in the dispersion-free case, that the perceptual function, P:W→X, that maps states of the external world W onto perceptual experiences X, is not veridical in the following two senses. First, X is not a subset of W, so that none of our perceptual experiences are literally true of the world. Second, P is not a homomorphism of any structures intrinsic to W, other than the event structure required for probability, so that no structural relationships among our perceptions are literally true of the world.

pg. 1489

HS&P specifically call out Gibson’s theory of direct perception as an example of a naïve realist strategy, and propose that as a result evolution would not have favored any organism trying to implement this strategy. If naïve realist perceptual mappings are not evolutionarily stable, then direct perception cannot possibly be a thing. Interface theory therefore makes a very strong existential claim specifically about direct perception, and so it is important to demonstrate that there is a strong defense. The rest of this paper articulates that defense.

The Objections to the Ecological Approach

Interface theory raises one novel objection⁴ to the ecological approach that needs to be addressed:

First, Gibson got evolution wrong: He claimed that evolution shapes veridical perceptions of those aspects of the world that have adaptive significance for us. Thus Gibson proposed naïve realism, not the interface theory … .

pg.1500

The objection actually comes in two parts; the evolutionary supremacy of non-veridical interfaces and the Invention of Symmetry theorem. I will tackle these separately, by first sketching out the defensive moves, and then defending these more fully in the following section.

Objection 1: The Evolutionary Supremacy of Non-Veridical Strategies

HS&P class the ecological approach as a “naïve realist” interface strategy, in which at least some of the structure of the world is preserved in perceptual experience. (The ecological solution is indeed this kind of realism, although as we shall see it is not an interface strategy.) HS&P then pit a full range of interface strategies, including naïve realism, against one another in evolutionary simulations, and find that

strict interface strategies that are tuned to fitness routinely drive naïve realist and critical realist strategies to extinction … .The only situation in which realists have a chance against interface strategies is when payoff varies monotonically with resource quantity, i.e., when truths and payoffs are roughly the same thing.

pg.1486

This happens because of the nature of the simulations; the payoff function literally rewards fitness and not veridicality, so when the two are not the same veridicality cannot win. HS&P defend this feature of the simulations by saying

… we cannot expect, in general, that payoff functions vary monotonically with truth, because (1) monotonic functions are a (unbiased) measure zero subset of the possible payoff functions, and (2) even if they weren’t, the ubiquitous biological need for homeostasis militates against them. Thus, we cannot expect, in general, that natural selection has tuned our perceptions to truth, i.e., we cannot expect our perceptions to be veridical.

pg.1486-7

Their homeostasis example is about “amount of water.” The payoff for water is not a monotonic linear function of amount of water; a little and a lot are both bad for you.

So fitness and veridicality are rarely the same thing and because evolution favors fitness, it will always out-compete veridicality. Gibson’s approach entails veridical perception and therefore he is out of the game.

Defense 1: Perception of the Ecologically Scaled World Tracks Payoff Directly

Recall that veridical mappings can survive the simulations “when payoff varies monotonically with resource quantity, i.e., when truths and payoffs are roughly the same thing” (pg. 1486). Payoff is the function that describes the world in behaviorally relevant terms – the consequences (for good or ill) of an amount of water, and not simply the amount of water. HS&P claim that the resource quantity and payoff functions are never the same because the world (as described by physics in terms of observer-independent properties; “truth”) is never the same as the world described in terms of behavioral consequences (“payoff”).

Gibson, however, was not trying to understand perception of the world as described by the science of physics. Physics takes an abstract, observer-independent position in order to understand the laws that operate at all spatial and temporal scales across the universe. It is, however, the wrong ontology to identify the objects of perception for an embodied observer living at a specific spatial and temporal scale. Gibson (1979) lays out an ecological-level physical analysis of the world-to-be-experienced by such an observer; in fact, he recognized that this was so important it is the first four chapters of this book on vision, prior to any discussion about eyes! The resulting ontology carves up the world so that behaviorally relevant properties (which Gibson terms affordances) are primary, and it hypothesizes that these properties are the objects of perception. To the extent, these objects create information about themselves, Gibson’s analysis predicts that they can be perceived directly, no inference required (Turvey et al., 1981).

In effect, Gibson’s ecological approach is a hypothesis that the ecological “truth” of the physical world is already in the form of the “payoff” function and that this can be perceived (this relates to the concept of the umwelt; Von Uexküll, 1957). I will review the details below (see “The Nature of the Ecologically Scaled World”); for now I just want to note that given the right ontology (world structure) to preserve, naïve realist perceptual systems like the ecological approach remain viable competitors in the evolutionary simulations. I would also note here, though, that the fact an interface theory analysis of affordances would find them to be both physical world structure and the interface contents is a hint that such an analysis is not the way to go.

However, even if we did evolve such an interface, HS&P have another immediate objection to it ever being sufficiently adaptive; the Invention of Symmetry theorem.

Objection 2: The Invention of Symmetry Theorem

Interface theory is formalized in terms of set theory. An interface is a mapping function from one set of things (the world as described by physics) to another (perceptual experience). HS&P have an argument, however, that shows that perceptual experience set structure actually tells you nothing about the world set structure. There is no necessary link, no symmetry, between them. This implies that the rich structure and sensible behavior of our perceptual experience (including things like objects and events) is actually not evidence that these things exist in the world set; they could still just be fictional icons in the interface. The argument is called the Invention of Symmetry Theorem, and its corollary, the Invention of Space-Time Theorem.

After a formal set theoretic definition (pg. 1498) they summarize it this way:

The … theorem shows that the world itself may not share any of the symmetries that the observer observes. The world need not have the structure the observer perceives, no matter how complex that structure is and no matter how predictably and systematically that structure transforms as the observer acts.

pg.1498

HS&P do not note this explicitly, but this is potentially quite a strong attack on the foundations of the ecological approach. As we shall see below (“The Nature of Ecological Information”) one key ecological hypothesis is that information can specify properties of the world, and specification rests on a symmetry principle. Information can be used to perceive the world because its structure is lawfully related to structure in the world, and detecting the former is therefore informative about the latter (Shaw and McIntyre & Mace, 1974; Turvey et al., 1981). If symmetries are actually all cognitive inventions and cannot support an observer “tracking back” from their perceptual experience to the world, then specification is not possible and ecological information cannot support perception or action.

Defense 2: Ecological Perception Discovers, Not Invents Symmetry

The important thing to remember about interface theory is that it is not a theory of physics; HS&P have not discovered anything new about the nature of the universe. It is a theory of perception, specifically an inferential, constructivist, representational theory of perception, so their claims are about how we perceive that universe. The Invention of Symmetry theorem therefore cannot possibly be claiming “it is physically impossible to track from any symmetrical mapping of the world back to the world,” it can only be claiming “it is psychologically impossible to track back from some symmetrical mappings (specifically, inferential interfaces) to the world.“ In other words, not all symmetries are invented; some exist due to the laws of physics, rather than being the creation of a perceptual interface.

Luckily for the ecological approach, ecological information is not a mental representation and therefore not an interface in which the relevant symmetry was invented. Ecological information is created by the operation of ecological-scale laws of physics, and it is in the world, waiting to be discovered and used by a perceptual system. Information is not constructed by us, its form is not optional, and the symmetry it has that perception depends on has not been created by a process of inference. As such, it is not within the (merely psychological) scope of the Invention of Symmetry Theorem.

In short, the Invention of Symmetry theorem applies only to the subset of possible symmetries that are invented via inference. The symmetry underpinning specification for ecological information is not in this subset. It is not a psychological invention, it is the result of an ecological level lawful physical process, and thus is safe from this problem.

Interfaces and Information Are Different In Kind – The Argument from Ontology

The previous section explicitly defended the ecological approach from the critiques HS&P level at the ecological approach, most importantly their strong claim that evolution would never favor a Gibsonian perceptual system. The thread that underpins the two defensive moves I made is the claim that the ecological approach has an entirely different ontology to representational, inferential approaches such as interface theory. It is now time to back that claim up with details.

This section will therefore first review the Establishment, extensional ontology of inferential theories of perception-like interface theory, and highlight the key unsolved problems that this ontology creates (representation grounding, representation selection). I will then review the ecological, intensional ontology that proposes a way to solve these problems by reframing the question of the nature of perception (Turvey et al., 1981; see also Chemero, 2009; Shaw & McIntyre 1974; Mace, 1977; Shaw et al., 1982; M. Turvey, 1977; Turvey, 2019; Turvey & Shaw, 1979). The goal of this section is to characterize the ontologies, identify that interface theory is only entailed by the Establishment ontology, and that the ecological ontology solves the problems without ever leading to interfaces or any of their consequences.

The Problems with Extensional Analyses of Perception

The Establishment ontology begins by characterizing the physical world-to-be-perceived in terms of observer-independent properties from physics (Fodor & Pylyshyn, 1981). The world as described by physics does not contain much (if any) of the content of perceptual experience, and so that content must be constructed via inference. In order to support adaptive behavior, that inference requires “concepts that stand for environmental properties and concepts that stand for how those properties structure energy distributions in media” (Turvey et al., pg.246). These concepts must, for a variety of reasons, be implemented as computational representational systems (Golonka & Wilson, 2019; Fodor, 1980; Newell, 1980; Pylyshyn, 1989).

Under this analysis, animals are therefore lawfully connected to the physical world, but are only connected to the behaviorally relevant properties of the world by inference – no law based connection. Another way to frame this is that behaviorally relevant properties (intensions) of things can only be perceived via inference based on the things implementing that property (extensions); it is an extensional analysis of perception (Turvey et al., 1981). More on this below.

This extensional ontology has two deep problems, however.

Representational Grounding

The first problem is that if you are only lawfully connected to extensions, there is no suitably constrained route to infer the correct intensions; you cannot build the adaptive conceptual systems you require if perception is not up to the task of linking you to behaviorally relevant properties to begin with. So where do these conceptual systems get their content, and how do they have the content that they have, if not via perception? This is the essence of the symbol grounding problem (Bickhard, 2009; Harnad, 1990; Searle, 1980) which remains a major unsolved problem at the core of Establishment cognitive science.

In order to be able to effectively evaluate whether a proposed representation has been successfully grounded, Bickhard (2009) frames the question as the need for representational content to be internally related⁵ to the thing it represents;

“ … an internal relation is internal in a sense of being necessary in order for one or more of the relata to be what it is. … if content is internally related to representation, then whatever constitutes such a representation cannot be what it is without bearing that content”

(pgs. 562-563).

When we apply this framing to interface theory, it is clear that interfaces and their contents are not related to each other in the necessary way to be grounded. Quite the contrary, interfaces are conceptual systems that explicitly have no obligation at all to be internally related to the world they represent. There is no lawful, necessary link between the form of icons in an interface and the thing they represent.

HS&P recognize that this problem is serious and propose a “Perception-Decision-Action” loop as a grounding mechanism (refer to Figure 1). W is the world set; G is the set of possible actions; X is the perceptual representation of W. An observer selects an action from G based on X (they make a decision, D), and this action, A, has an effect on W. That change in W is then perceived by X via perceptual channel P.

Figure 1. The Perception-Decision-Action loop (adapted from Hoffman et al., 2015)

Interface theory imposes a constraint that has to be addressed. Our observer cannot know W (because of the Invention of Symmetry Problem), which means it can have no knowledge of either A or P. All it can know is that it made a decision D and that eventually led to a change in X that either improves or reduces the fitness of future decisions; it is the overall PDA loop that creates information in X about fitness, which is what X is trying to represent. HS&P then propose that organisms can simply tweak D (over either evolutionary or individual lifespan timescales) and hunt for optimal fitness in the resulting change in X. They propose that this system is grounded via the looping interaction with the world.

This idea is close in spirit to Bickhard’s (2009) interactivist model, his preferred solution to the need for internal relations to ground representations, so it is a solid swing at a solution. But fitness remains a problematic relation; for example, in the PDA loop, fitness is evaluated entirely within the representation X, but with reference to what? There seems to be a need for an “interpreter” that is external to the representation X, and Bickhard notes this immediately becomes an “infinite regress of interpreters interpreting” (p 573) if the interpreter is not themselves grounded. The problem stems from the fact that there is no necessary connection between the form of an interface and the form of the world to be represented, and grounding requires a law-based link from world to perception (Harnad, 1990). Interface theory (like all inferential theories⁶) therefore still owes us an account of where representational content comes from.

Which Representation?

Even if you could build an inferential system that had the appropriate concepts, Turvey et al. (1981) show that if perception begins with extensions, there is no way to constrain which representation is deployed at any given moment in time in order to get to the behaviorally relevant, intensional property. As an example, they discuss the marsh periwinkle, and a problem that it faces.

In the periwinkle’s environment, there are things (plant stems) that have the property “can be climbed on.” The plant stems are extensions (specific implementations) of the intensional property “can be climbed on.” There are also things (plant stems, other snails) that have the property “are an obstacle to locomotion.” The plant stems and the snails are extensions of this property. Organisms need access to the intensional properties, and not just the extension, in order to produce adaptive behavior, and the Establishment ontology asserts that organisms are only connected to intensions via inferences on their extensions. But in the example above, the extensions of the two properties overlap. If you detect a plant stem, you don’t know which property you’re dealing with. Is it currently a climbing opportunity, or an obstacle? There is no way to reliably guide inference to the correct (or most adaptive) belief about properties of plant stems. Interface theory is very explicitly committed to this analysis; interfaces are literally mappings from perceptual experience to intensional properties via inference on extensions of those properties. HS&P offer no solution to this problem.

Because these problems are so serious and intractable, Turvey et al. (1981) conclude that perception, if it is going to work, must be directly of intensional properties, and so our theories must therefore begin with an intensional analysis of what there is to be perceived. The ecological approach (Gibson, 1979) is such an analysis, making it different in kind to interface theory. The following two sections detail this different ontologies, in order to make it clear that the results of the extensional interface theory can have no bearing on the viability of the intensional ecological approach.

The Nature of the Ecologically Scaled World

HS&P note that “The only situation in which realists have a chance against interface strategies is when payoff varies monotonically with resource quantity, i.e., when truths and payoffs are roughly the same thing.” (pg.1486). My first defensive move was to state that veridical perception of the ecologically scaled physical world is such a case. This section justifies this move.

Interface theory assumes that the appropriate way to describe the world-to-be-perceived is through the lens of the science of physics. The problem here is that physics describes an abstract, observer independent world, shaped by laws with near-universal scope, from the very small to the very large, to the very slow to the very fast. The laws apply across a wide range of conditions, making physics the powerful science that it is today. Biological organisms, however, only operate within a fairly narrow range of conditions. We are medium sized objects moving at fairly slow speeds at fairly sedate temperatures, and our biology is intimately connected to the more limited set of laws whose scope is only the spatial and temporal scales we operate at. This is okay; however, laws do not require universal scope to be laws, and in fact for this reason the scope is part of the specification of any law. In order to understand the physical context in which perceptual systems evolve, our theories of perception therefore need to begin with an ecologically scaled physical description of the world-to-be-perceived and the laws that dominate physical processes at that scale.

At the ecological scale, the environment to be perceived is best described at the level of Newtonian mechanics using the formalisms of dynamical systems theory (DST; Bingham, 1995; Warren, 2006; Wilson & Bingham, 2001). Specifically, a complete formal description of the world-to-be-perceived requires reference to the medium-sized states of the world, how they change over medium timescales, and the not-especially-extreme forces that caused those changes. This complete description allows units of time, position and its temporal derivatives, and mass.

DST is the right language, but it is not intrinsically a theory of the ecological scale, nor is it a theory of behavior. There is nothing in DST that can guide you to the behaviorally relevant properties of the environment (the correct “objects of perception”); it is just the right way to formalize those properties once they have been identified by a theoretically driven empirical programme of psychological research. The ecological approach is just such a programme (Golonka & Wilson, 2012) and it proposes that the most psychologically relevant set of physical properties in the ecologically scaled environment are affordances (Gibson, 1979; Scarantino, 2003; Turvey, 1992; Turvey et al., 1981).

Affordances are dispositional (Mumford, 2003) intensional properties of the world⁷; specifically, they are the subset of these properties that are action-relevant and perceptible. To say “My coffee mug affords grasping” is to identify that the relations between the surfaces that constitute the mug make it so that when certain satisfying conditions are met (say, the presence of an appropriately sized hand) the disposition to be grasped can be manifested. The affordance disposition is constituted by the cup’s surfaces and their physical properties (the anchoring properties of the disposition; Turvey et al., 1981) but the affordance itself is also a distinct higher-order property of the cup. The ecological hypothesis is that perceiving-acting organisms organize their behavior with respect to this latter property directly, and not via internal, inferential combination of the various anchoring properties.

No physicist would ever identify affordance properties, but this is not because they are not real. Rather, it is because the physicist does not need to carve the world along these joints in order to explain the behavior of, say, a projectile flying through the air. Perceiving-acting organisms, in turn, do not need to carve the world up the way physicists do; instead, they must learn to pick out properties like affordances because those are the ones required to coordinate and control their behavior. But even though affordances are not properties predicted by a theory of physics, they are still physically real properties and can be described as such, using the language of DST (see Wilson et al., 2016a, 2016b, 2018 for a detailed description of this research programme in the context of long-distance throwing). Just as biology and chemistry do not reduce neatly to physics, neither does psychology, and we must take this seriously. Our theories of perception must explain the perception of the organism’s world, and not the physicist’s.

Interestingly, this analysis provides an alternative way to read and interpret the results of HS&P’s simulations. They repeatedly find that perceptual mappings that veridically preserve the structure of the world as described by physics are out-evolved because those properties cannot directly support adaptive, fit behavior. They are beaten by interfaces that directly represent the world in terms of behaviorally relevant properties. The ecological hypothesis is simply that these latter properties of the world exist prior to perception and do not need to be inferred. The straight-forward prediction would be that perceptual mappings that veridically preserve the structure of the world as described by ecological physics (e.g., affordances) should be able to directly support adaptive, fit behavior and survive the simulations. HS&P have not tested this exact hypothesis in their simulations, but the fact that naïve realist strategies that track fitness do survive suggests it should work out in our favor. If it did, this would imply that veridical mapping functions are only failing because HS&P have mischaracterized the physical world.

The Nature of Ecological Information

The second objection to the ecological approach is the Invention of Symmetry problem. My defensive move there was to point out that the symmetry underpinning specification falls outside the psychological scope of the problem. This section justifies that move by explaining how ecological information is formed and used.

The ecological world is a dynamical place. As environmental dynamics unfold over time, the various components of the dynamic (such as surfaces) interact with various energy media; light bounces of them, they generate sound waves in the atmosphere, and so on. The energy that reflects off a surface is lawfully changed by that interaction, from a field of unstructured, symmetrical, radiant energy into the structured, asymmetrical, ambient energy called an array; into ecological information for perception. The structure in the array is then available at a distance from the dynamical property that caused it, enabling perception of that property from a distance.

The structure of these arrays can be completely captured at the level of kinematics; time and motions, but not forces. The kinematic structure of that information is lawfully linked to the underlying dynamics by the ecological-scale laws that govern the projection of world into energy (Turvey et al., 1981), but that structure cannot be identical to the underlying dynamics because it is a projection into a medium that can only support kinematic structure (the “perceptual bottleneck”; Bingham, 1988). Fortunately, information can still specify (map 1:1 to) those dynamical properties, and this specification can be used as direct access to behaviorally relevant properties by organisms because the mapping is between information and those intensional properties, and not the extensional things that carry those properties. The basis of perception is therefore the kinematic specification of dynamics (Runeson & Frykholm, 1983).

Specification can be formalized as a symmetry principle (Shaw et al., 1974; Turvey et al., 1981; see Chemero, 2009, Chapter 6 for the excellent, clear review from which I am borrowing here). The way the world is lawfully specifies that information is the way it is, which in turn lawfully specifies that perceptual experience is what it is; this then means that perceptual experience being what it is lawfully specifies what the information is, and what the information is lawfully specifies the way the world is. This symmetry is the underlying mechanism that allows perception to be direct and adaptive. But this symmetry is clearly outside the scope of the Invention of Symmetry theorem; it is not being psychologically inferred, it is a result of the operation of ecological scale physical laws. Critically, perceptual systems do not contain any knowledge of or assumptions about the underlying symmetry principle; organisms simply organize their behavior with respect to information variables and this works as access to behaviorally relevant properties (i.e. “is fit”) to the extent that the symmetry actually holds (see Golonka, 2015,; Golonka & Wilson, 2019a, b for more detail on this point).

Ecologically Solving Establishment Problems

The above analysis of the ecological world and the information it creates provides solutions to all the various problems posed by Establishment theories.

First, ecological information solves the grounding problem. According to Harnad (1990), the symbol grounding problem is eliminated if the relationship between the world and the medium onto which it is projected is entirely lawful (meaning, non-arbitrary) because higher level systems inherit the lower level grounding. This is the same as making the relation between the world and information an internal one (Bickhard, 2009). Ecological information is created via the operation of ecologically scaled laws (Turvey et al., 1981) and thus is grounded (Golonka & Wilson, 2019a).

Second, ecological information solves the representation selection question, in two ways. First, the ambiguity underlying this problem only occurs if perception is of extensions of properties; information is specific to the intensional properties themselves and so no ambiguity arises. Second, because information is created by the interaction of the current task dynamics with various energy media, only information about that dynamic is available in the first place, so the organism does not need to choose how to interpret that information.

Third, the fact that information is specific to intensional properties, and not to propertied things (the extensions) leads to a powerful reframing of the issue of misperception. HS&P discuss several examples (pg. 1481) of animals adaptively using what seem to be non-veridical perceptual mappings. For example, dragonflies need to lay their eggs by water but do not seem to perceive “water” per se; instead, they pick up a pattern of polarized light caused in their niche by water and lay their eggs at the source of that light. Their perceptual experience is being structured by “polarized light,” which is not identical to “water” but supports an inference of the presence of “water”; their experience is non-veridical, but still adaptive. The problem is that “polarized light” does not specify “water”; shiny tombstones can produce the same effect, which can then lead dragonflies to make serious errors in where they lay their eggs. For HS&P, this error reveals both the non-veridical mapping at work and the presence of inference in the perceptual process.

The ecological solution comes in two parts. First, because information is about properties (intensions), not the bearers of those properties (extensions), dragonflies are not non-veridically perceiving water, they are veridically perceiving the presence of a surface with a particular property that polarizes light in a particular way. In the niche of the dragonfly, this polarized light means that the surface supports egg laying (because it is water) and the dragonfly learns this. This relation is a lawful one with a scope of the dragonfly’s niche; a small scope, but one well suited to the dragonfly. The presence of a shiny tomb breaks the scope of the law; however, it is an intrusion (in this case by human manufacturing) into the niche of the dragonfly, which breaks the lawfulness of the relation the dragonfly’s perception had depended on. This of course then leads to a maladaptive behavior; but Turvey et al. (1981) argue extensively that from the first person perspective of the dragonfly⁸ there is no misperception and the behavior is not an error. The polarized light really is there and it has been correctly detected by the dragonfly. Within the niche of the dragonfly “polarized light” is lawfully caused by “water” and so the dragonfly would therefore only be in error if it did not try to lay eggs on that surface.

What we actually learn from misperception or illusion examples is that ecological laws have scope; they are not universal. This is true of all laws; even laws with enormous scope such as the law of gravity break down, for example, in a singularity. The scope of ecological laws are simply smaller and easier to cross. Perception relies on those laws in order to be adaptive, and so of course behavior becomes maladaptive if the scope is violated. Given time, however, and the presence of an information variable that distinguishes between some property of water and tombs, the niche of the dragonfly will simply expand to accommodate the change.⁹

In summary, the ecological ontology is that organisms veridically, directly perceive behaviorally relevant dynamical properties (intensions) of the world by virtue of the ecological law-based symmetry between those properties and the information they create that allows the information to specify the property that created it. This detailed framework is the analysis that supports my defense of the ecological approach from the challenges posed by interface theory.

Conclusion

Interface theory makes many bold claims. Perceptual experience is an adaptive lie, it says, like the virtual desktop environment of a computer operating system. This is because in order to be evolutionarily fit, organisms must shape their behavior with respect to behaviorally relevant properties of the world, but the world as described by physics just isn’t carved up this way. Organisms must infer their way to the behaviorally relevant ones; they must invent a symmetry to a world that does not quite exist the way it looks in perceptual experience. In short, we do not perceive the world as it is.

The first thing to note is that this analysis is not actually as radical as it seems. It is, in fact, just a crystal clear expression of the inferential Establishment view of perception as inference on ambiguous sensory data. If behaviorally relevant properties of the world must be inferred because they cannot be specified in sensory data, of course the resulting mental representation shaping perceptual experience will not resemble the world that caused the sensory data in the first place. In this context, the “Invention of Symmetry” problem is revealed to actually be the representation/symbol grounding problem, and it is indeed as much of an issue for inference-based theories of perception as HS&P say. It has largely stopped being discussed, however, so while not new, interface theory has done the field a great service by formalizing the Establishment hypothesis in such impressive and inescapable detail such that grounding is forced back into the discussion. Interface theory embraces, rather than solves the grounding problem, however, which is what leads to all the worrying implications.

HS&P make one wrong move, however; they apply their analysis to the ecological approach to perception and action and conclude that such a perceptual system could never evolve. What they have failed to notice is that the ecological approach is not simply proposing a naïve realist interface mapping. It is actually the hypothesis that such interfaces are not required at all because perception can, in fact, be directly of behaviorally relevant properties of the world. This properly radical idea becomes possible when we stop trying to explain the perception of the observer-independent world of physics, and instead work to explain the perception of the ecological umwelt of the organism (Baggs & Chemero, 2019). The ecological approach begins by correctly characterizing the new ontology of that ecological world (Gibson, 1979; Turvey et al., 1981) and then scientifically investigating the perception of those dynamical task properties (affordances) via the specifying information those properties lawfully create.

The inferential and ecological approaches therefore diverge right at the ontological base, and, unlike Establishment theories, the ecological approach is safely shielded from any implications of interface theory. It therefore remains a viable (and promising) hypothesis about the nature of our perceptual contact with the world.

Disclosure of potential conflicts of interest

No potential conflict of interest was reported by the author(s).

Acknowledgments

I would like to thank Dr Agnes Henson and Dr Sabrina Golonka for their advice and comments on this paper.

Additional information

Notes on contributors

Andrew D Wilson

Andrew D. Wilson is a Reader in Psychology at Leeds Beckett University. He studies the perceptual control of action and writes extensively on the ecological approach to cognitive science

Notes

1. I’ve borrowed this phrase from Fodor and Pylyshyn (1981) because it’s a useful collective name for the various mainstream theories of perception, such as interface theory, that implement an inferential, representational approach.

2. A note for interested readers; Putnam (1980) produced a mathematical argument along the same lines as interface theory and ending in the same place. However, in typical Putnam style, he kept thinking about the issue, decided he had made an error and turned instead toward a realist stance, more along the lines of Gibson (Putnam, 1994). I’m focused here on the specific ecological defense, but Putnam’s analyses may be another line of critical analysis of interface theory.

3. There may be reasons to disagree with the following labels; however, I will rely on HS&P’s terminology for the remainder of the paper to keep my rebuttal focused.

4. HS&P also raise two less novel objections. First, they claim that the ecological approach has no way to explain illusions. This is false (Turvey et al., 1981; De Wit et al., 2015) and there are examples of ecological accounts of apparent misperceptions (Runeson, 1988; Zhu & Bingham, 2011; Zhu et al., 2013). Second, they claim that the evidence for information processing accounts of perception is now overwhelming. While there are many papers on this, there is also a strong empirical base for direct perception accounts, both in behavioral work and more recently in neuroscience work (e.g., Van Der Meer et al., 2012, Agyei et al, 2016a, 2016b). In addition, when direct perception and information processing accounts are directly pitted against one another, the former prevails (e.g., Fink, Foo & Warren, 2009; Markkula et al., 2014; Mon-Williams & Bingham, 2008; Zaal & Bootsma, 2011; Zhu & Bingham, 2010). These are common objections to the ecological approach, but they are rooted in misconceptions of what Gibson was claiming and do not reflect the state of the research programme (Costall & Morris, 2015)

5. This is similar to Adams and Aizawa (2008) ‘mark of the cognitive’.

6. Bickhard (2009) also applies his analysis to the major modern players in the representational literature, specifically Millikan, Dretske, Fodor, and Cummins and shows it holds for each of these; see also Ramsey (2007) for related critiques.

7. There remains some debate in the ecological literature about how best to formalize affordances (e.g., Chemero, 2003, 2009; Rietveld & Kiverstein, 2014; Stoffregen, 2003) but the original dispositional analysis is the only one that explains how affordances create information and therefore be perceptible. For this reason the dispositional account remains the most ontologically appropriate (see Wilson, 2021, for details on this argument).

8. See (Barrett, 2011) for a detailed discussion about the importance of this perspective to the analysis of perception

9. This part of the learning process is called the education of attention; see Jacobs & Michaels, 2007 for a detailed analysis