The minority report: some common assumptions to reconsider in the modelling of the brain and behaviour

Abstract

Reverse-engineering the brain involves adopting and testing a hierarchy of working hypotheses regarding the computational problems that it solves, the representations and algorithms that it employs and the manner in which these are implemented. Because problem-level assumptions set the course for the entire research programme, it is particularly important to be open to the possibility that we have them wrong, but tacit algorithm- and implementation-level hypotheses can also benefit from occasional scrutiny. This paper focuses on the extent to which our computational understanding of how the brain works is shaped by three such rarely discussed assumptions, which span the levels of Marr's hierarchy: (i) that animal behaviour amounts to a series of stimulus/response bouts, (ii) that learning can be adequately modelled as being driven by the optimisation of a fixed objective function and (iii) that massively parallel, uniformly connected layered or recurrent network architectures suffice to support learning and behaviour. In comparison, a more realistic approach acknowledges that animal behaviour in the wild is characterised by dynamically branching serial order and is often agentic rather than reactive. Arguably, such behaviour calls for open-ended learning of world structure and may require a neural architecture that includes precisely wired circuits reflecting the serial and branching structure of behavioural tasks.

Keywords::

• complex sequential behaviour
• deep networks
• reinforcement learning
• mammalian brain architecture

1. Introduction

David Marr's posthumous classic Vision (Marr, 1982) began famously with a question: ‘What does it mean, to see?’ Putting such questions up front and centre contributed at the time to a conceptual realignment in the study of natural and artificial neural computation, following which it became accepted that understanding the nature of a problem in cognition is part and parcel of understanding its possible solutions, as well as the actual solution implemented by the brain and used by it to decide how to behave.

But what does it mean, to behave? What computational problems do animals routinely face and solve? Problems that arise ‘in the wild’ are not revealed as such to cognitive scientists (even in the laboratory, what goes on in the subjects’ heads may be not quite what the experimenter intends). To arrive at a reasonable formulation of these problems, we have no choice but to put forward and evaluate hypotheses. These are based on the animal's range of behaviours, which in turn are interdependent with its ecological niche, social habits, body mechanics and physiology, and neural architecture.

Insofar as our hypotheses regarding the problems that animals ostensibly solve are based on abductive reasoning, they are subject to biases. Some of these biases are entirely of our own making. For instance, if an effective algorithmic framework happens to become available for a class of problems, these then tend to loom larger as possible factors in theories of behaviour.1 As a quick example (to be revisited later), consider the widespread influence exerted on cognitive science by the invention of the back-propagation algorithm for training multilayer perceptrons (Rumelhart, Hinton, & Williams, 1986). This made it possible to use a relatively simple (layered and uniformly connected) ‘neural network’ architecture to learn a large and important class of functions for mapping inputs to outputs (Cybenko, 1989; Haussler, 1989; Hartman, Keeler, & Kowalski, 1990; Hornik, Stinchcombe, & White, 1989). The resulting spate of connectionist approaches to various cognitive tasks, including visual perception (Mel, 1991) and motor control (Mussa-Ivaldi & Giszter, 1992), naturally involved the assumption that such tasks can be adequately reduced to mapping inputs to outputs. While rarely stated or defended explicitly (as it was in (Poggio, 1990)), this assumption grew to be ubiquitous, quickly spreading even to theories of sequential behaviours such as language (Elman, 1990).

To guard against such biases, it makes sense occasionally for us to pause and reevaluate our working hypotheses, including any tacit assumptions, in the broader context of understanding real brains and real behaviour. This paper undertakes such a reevaluation for issues that extend over the three levels of the methodological hierarchy defined by Marr and Poggio (1977; cf. Marr, 1982): problem, representation/algorithm and implementation. On each of these levels, I offer a critical look at some common and far-reaching assumptions that are yet rarely discussed.

The remainder of the paper is structured as follows. In Section 2, I discuss the notion that the control of behaviour can be effectively mastered by learning a (possibly very complex) input–output mapping and argue that it is too limited to account for realistic animal behaviour, which is sequential, hierarchically structured, dynamically branching and agentic. Section 3 proposes that the common view of learning as the optimisation of a fixed cost function (over a possibly very complex input/output or state × action space) is likewise too limited to support important sequential behaviours, such as language. In Section 4, I contend that the gross architecture of the mammalian brain can be better understood if we accept that certain tasks in behavioural learning and control are best served not by the common, uniformly connected multilayer networks, but rather by precisely wired circuits, whose topology reflects the connectivity structure of the task space. Section 5 recapitulates the proposed ideas and illustrates them on a case study of a computational model of language acquisition and translation. Finally, Section 6 offers a summary and concluding remarks.2

2. Behaviour

Parts of this section will appear to reprise a debate that many of us in psychology and neuroscience believe has been long resolved and put to rest: the debate between the behaviourist and situationist approaches on the one hand and interactionist and cognitive approaches on the other (Bowers, 1973; Neisser, 1967). My intention here is, however, not to revive that debate, but rather to illustrate the extent to which behaviour is still tacitly treated as a series of responses to stimuli, even within theoretical frameworks that are otherwise avowedly cognitive and explicitly computational.

2.1. The S/R assumption

In psychology, the conception of behaviour as a series of responses to stimuli – what I shall call the S/R assumption – appears already in William James (1911, quoted in Phillips, 1971):

The structural unit of the nervous system is in fact a triad, neither of whose elements has any independent existence. The sensory impression exists only for the sake of awaking the central process of reflection, and the central process of reflection exists only for the sake of calling forth the final act.

Note how the S/R assumption here is back-grounded: James's real concern was with understanding the function of the nervous system, which he took to be the mediation between stimuli and responses, without questioning the very possibility of reducing behaviour to a series of responses to stimuli.

Fifty years later, the S/R assumption still reigned in physiological psychology: the subtitle on the cover of the 1961 Science Editions paperback printing of Donald Hebb's The Organization of Behavior (Hebb, 1949) read ‘Stimulus and response – and what occurs in the brain in the interval between them’ (Edelman, 2012, p. 1121). Even the subsequent cognitive turn in psychology and neuroscience had little effect on the S/R view of behaviour, which ‘appeals to many psychologists because it seems to be an explicitly causal analysis’ (Bowers, 1973, p. 309).

This appeal is still strong. As recently as 2003, a handbook of experimental psychology stated that ‘Learning, like many other central concepts in psychology, is referred to as an intervening variable, that is, an unobservable construct postulated to account for stimulus-response regularities’ (Papini, 2003, p. 231). Bowers (1973, p. 316) anticipated such persistence of behaviourism, attributing it to the continued influence of the S/R assumption: ‘So long as cognition is viewed primarily as a response mediating the causal impact of external stimuli, it cannot easily be viewed as initiating, maintaining, or explaining behaviour.’

All this might have been just as well, if the reliance on the S/R assumption in psychology and brain research had been more productive. Arguably, however, it has instead impeded progress. In a recent personal retrospective on several decades of studying the nervous system of the fruit fly, Heisenberg (2014, p. 389) offers the following diagnosis of what he perceives to be a general impasse in this regard: ‘What is the problem with brain research? The problem is the input-output doctrine. It is the wrong dogma, the red herring.’

2.2. A machine learning perspective on the S/R doctrine: function approximation and deep networks

Importantly, the problem with the S/R assumption, or the ‘input-output doctrine’, is not the lack of theoretical understanding of, or practical solutions for, mapping inputs to outputs. On the contrary, as suggested in the introduction, it may be the relative conceptual clarity and computational ease of learning S/R mapping that contributes to its continued dominance in experimental psychology and neuroscience.

In machine learning and artificial intelligence, the problem of mapping stimuli to responses corresponds to function approximation – a well-understood task, which has been extensively studied and for which there exist provably effective algorithmic solutions, amenable to implementation by networks of simple neuron-like computing elements (Hornik et al., 1989; Poggio, 1990). Among the latter, deep network (DN) architectures have recently been gaining prominence (and media attention) by outperforming competing approaches on many popular benchmarks (Schmidhuber, 2015).

Briefly, DNs are uniform, multilayer neural networks that undergo supervised training to serve as generative models.3 Hinton (2007) summarises their properties by noting that adopting the generative approach reduces the need for labelled data (although given the large size of current models, very large amounts of data are still required), that many hidden layers can be stacked as long as learning is carried out one layer at a time (using Restricted Boltzmann Machines) and that the resulting model can be fine-tuned so as to boost its generative or discriminative performance. Many additional improvements to this general blueprint are possible (e.g. convolutional feature extraction (LeCun & Bengio, 1995) and extension to sequential data (Sutskever & Hinton, 2007)); see (Hinton, 2009) for a concise overview, (Bengio, 2009) for a tutorial introduction and (Schmidhuber, 2015) for an exhaustive history and references.

Networks with many nonlinear stages are extremely powerful, in that they can express functional relationships of great complexity, in high-dimensional representation spaces. Such power comes at a price, especially in deep, uniform architectures, which present a particularly acute case of the fundamental credit assignment problem (Minsky, 1961): how to use supervision data to identify units responsible for each outcome and to modify their parameters to improve performance (Schmidhuber, 2015). With the discovery of effective training methods for DNs, it became clear that, for a wide variety of tasks, they are not only much better than traditional networks with one hidden layer, trained by back-propagation (Rumelhart et al., 1986), but are also in fact capable of outperforming – on those tasks – any available learning-based method (Schmidhuber, 2015).

It is here that the allure of DNs for psychology and neuroscience becomes clear: just as the S/R assumption causes the attention in those disciplines to focus on mapping inputs to outputs (at the expense of a broader conception of behaviour), mapping inputs to outputs is precisely the category of tasks on which DNs excel. As Schmidhuber (2015) concludes in his comprehensive review, what DNs do is ‘learn to perceive/encode/predict/classify patterns or pattern sequences’. It may seem surprising that this category of tasks includes many that arise in language, which involves complex sequential and structural dependencies that extend beyond S/R contingencies (Chomsky, 1959). This quandary is resolved if one considers closely the kinds of language task on which DNs do well (e.g. Socher, Lin, Ng, & Manning, 2011; Sutskever & Hinton, 2007; Sutskever, Vinyals, & Le, 2014). Invariably, these involve the mapping of entire structures to entire structures (as in learning to choose the best parse tree for a given input sentence). In other words, DNs can serve as a good model of behaviour only if we pretend that behaviour amounts to one Google (or Watson) query after another.

2.3. A broader view of behaviour: agency, hierarchy, situated dynamics

The view of behaviour as an arc from stimulus to response (perhaps via some thinking or intervening nervous activity) is, however, untenable. To begin with, the problematicity of the very concepts of stimulus and response, discussed at length by Chomsky (1959), had been occasionally acknowledged by B.F. Skinner himself, who complained that ‘It is very difficult to find a stimulus and response which maintain precisely the same properties upon two successive occasions’ (Skinner, 1935, quoted in Glenn, Ellis, & Greenspoon, 1992). Likewise, in a generally very favourable review of Skinner's legacy, Epstein (1991, p. 362) noted that the concepts of stimulus and response may not apply because behaviour is really continuous: ‘The click of a microswitch suggests, falsely, that a discrete “response” has occurred, but the rat is active continuously, and what occurs is multidimensional and complex.’

Furthermore, behaviour is often agentic or animal-initiated, in a manner that blurs the distinction between a stimulus, which could in reality be an act on the part of the agent, and a response, which could be the resulting snapshot of the environment. Thus, long before Chomsky's (1959) critique of Skinner's attempt to extend the S/R approach to language, John Dewey (1896) wrote:

What we have is a circuit, not an arc or broken segment of a circle. […] The motor response determines the stimulus, just as truly as sensory stimulus determines movement. […] There is simply a continuously ordered sequence of acts, all adapted in themselves and in the order of their sequence, to reach a certain objective end, the reproduction of the species, the preservation of life, locomotion to a certain place. The end has got thoroughly organized into the means.

Revisiting this idea in later years, Dewey (1931, p. 255) gave it an even stronger formulation:

No external change is a stimulus in and of itself. It becomes the stimulus in virtue of what the organism is already preoccupied with. To call it, to think of it, as a stimulus without taking into account the behaviour that is already going on is so arbitrary as to be nonsensical.

Lashley (1951) concurred, noting that the active role of the animal in shaping its behaviour implies that ‘the input is never into a quiescent or static system, but always into a system which is already actively excited and organized’.4 Furthermore, Lashley (1951, p. 123) argued, behaviour is serial and, moreover, its control by the brain cannot be a matter of a simply stringing together a series of independent acts or responses:

Certainly language presents in a most striking form the integrative functions that are characteristic of the cerebral cortex and that reach their highest development in human thought processes. Temporal integration is not found exclusively in language; the coordination of leg movements in insects, the song of birds, the control of trotting and pacing in a gaited horse, the rat running the maze, the architect designing a house, and the carpenter sawing a board present a problem of sequences of action which cannot be explained in terms of successions of external stimuli.

Rather, the control of such behaviours must be hierarchical – a view that has come to enjoy broad empirical support with regard both to language in humans and to other serial behaviours in humans and other animals (Houghton & Hartley, 1996; Rosenbaum, Cohen, Jax, Weiss, & van der Wel, 2007).

In addition to exhibiting a sequential and hierarchical structure, linguistic behaviour is situated and dynamic: speakers and listeners refer to and rely on their physical settings and coordinate their parts of the conversation in an ongoing manner (e.g. Crocker, Knoeferle, & Mayberry, 2010; Dale, Fusaroli, Duran, & Richardson, 2013; Spivey & Dale, 2006). Lessons from ethology, where the S/R approach has been less dominant,5 suggest that these characteristics are typical of animal sequential behaviour in general.

As an example of a common type of behaviour that is situated, sequentially and hierarchically structured, and in many cases dynamically socially coordinated, we may consider foraging (Galef & Giraldeau, 2001; Kamil & Sargent, 1981). Many foraging species, including mammals, exhibit the kind of cognitive flexibility and hierarchical real-time control (e.g. Maaswinkel & Whishaw, 1999; Sallabanks, 1993) that are unlikely to be reducible to S/R learning. For instance, in goat foraging one finds ‘a mixture of influences across a range of spatial and temporal scales’ (Derry, 2004, p. 140). More generally, in foraging ‘choices between simultaneously offered goods are relatively rare and foraging is instead dominated by sequential choices between exploiting nearby resources versus exploring elsewhere’ (Cisek, 2012, p. 930). Human foraging, in particular, involves complex, dynamic decision-making (Pacheco-Cobos, Rosetti, Cuatianquiz, & Hudson, 2010). When foraging is carried out socially, in groups of several people, it is, as one would expect, facilitated by verbal communication (King et al., 2011). Indeed, the behavioural needs arising in foraging have been posited as one of the sources of evolutionary pressure that led to the emergence of language (Cangelosi, 2001; Kolodny, Edelman, & Lotem, 2014).

2.4. The overarching task in complex behaviour

Given that realistic behaviour is agentic, sequentially and hierarchically structured, situated and dynamic, what tasks does it give rise to? In light of the preceding discussion, we would do well to raise our sights and consider the idea that the overarching problem in managing behaviour is deciding what to do next: ‘Briefly, one of the basic problems that must be solved is to resolve which functional system, at any point in time, should be permitted to direct “the final common motor path”, i.e. determine behavioural output’ (Redgrave, Gurney, Stafford, Thirkettle, & Lewis, 2013, p. 130).6

For a realistic behaviour, the problem space associated with deciding what to do next will typically be large, complex, diachronic and open-ended. It is large because of the combinatorics arising from the sequential nature of behaviour and the multiple actions choices available at each step (think of the number of motor degrees of freedom in a typical vertebrate body plan). It is complex because of the interactions among the relevant contextual factors, both external and internal or agentic. It is diachronic because contextual effects extend – and may change – over time. And it is open-ended because of novelty, which can also be external, as in a novel physical or social situation, or internal, as when an opportunity for a novel action arises and needs to be considered.

It is the diachronic effects and open-endedness that are particularly troublesome for the S/R view of behaviour. Action choice typically cannot be reduced to a set of fixed S/R associations or rules; hence the common designation of rigid rule-following behaviours as ‘slavish’ or ‘robotic’ and the notion that ‘rules are there to be broken’. Nor can each action decision be the outcome of choosing a response to an immediately preceding stimulus. These challenges are especially obvious in human behaviour. In the use of language, in particular, the combinatorial space of action choices is both very large (due to the tens of thousands of lexical degrees of freedom) and very complex (due to the intricate syntactic constraints that govern word combination). This complexity, moreover, is not confined to the present moment in time, as a stimulus in the S/R framework is, but rather are spread over time and across multiple time scales: the next word in an utterance may depend in principle on the speaker's entire past life experience, as well as on another word choice that is still in the future.

3. Learning

Setting aside the S/R assumption in favour of the more general, yet arguably more useful, view of the key problem of behaviour as deciding what to do next, we may now descend one level in the Marr hierarchy (keeping in mind, as always, that the levels are interdependent) and ask what algorithms and representations would work for it. Approaching this question rationally (in the sense of Chater (2009)), we may assume that an agent situated in an initially unknown and possibly changing environment would do well to take into account the outcomes of its actions – an assumption that leads to the idea of reinforcement learning (Barto, 2013; Sutton & Barto, 1981, 1998; Woergoetter & Porr, 2007).

Reinforcement learning (RL) is an extremely general formalism: Schmidhuber (2015) points out that ‘any task with a computable description can be formulated in the RL framework.’ As such, making it explicit enough to serve as a model of animal behaviour (or an engine of robot behaviour), as well as understanding how it may work in the brain, requires making working hypotheses, or assumptions, about the representations involved and about the manner whereby these are modified by experience. In this section, I examine two types of such assumptions on which RL algorithms are based: those that have to do with what is to be represented and those that pertain to how outcomes affect the representations. I then explore the possible conceptual and practical benefits of modifying these assumptions.

3.1. Common assumptions in RL research

The conceptual roots of RL can be traced back to Thorndike's (1911) Law of Effect, according to which animals tend to repeat actions that are reinforced – followed by positive reward – and refrain from actions that are followed by negative reward or punishment (Barto, 2013; Chater, 2009). A straightforward formulation of RL thus makes the following assumptions regarding the ‘what’ and the ‘how’ design questions, respectively: (i) representing actions along with their expected rewards in various situations and (ii) choosing at each step the action such that the resulting sequential behaviour maximises cumulative reward (Woergoetter & Porr, 2007).

In model-based RL (Doll, Simon, & Daw, 2012), the agent builds and maintains a model or representation of the environment, which is used to predict the reward that would ensue from various actions, given the present state of affairs of the agent and of its environment. Such a model typically consists of two components: a mapping that associates state-action pairs with immediate utilities (used to evaluate present reward and to predict future reward from past history) and a mapping from state–action pairs to new states (Chater, 2009). The internal model may also focus exclusively on reward structure, without attempting to represent the state of the environment (Nakahara & Hikosaka, 2012). In model-free RL, in comparison, reward maximisation is attempted by mapping environment states directly to actions. Furthermore, in either model-free or model-based RL, hierarchical approaches impose structure on behaviour by grouping certain sequences of actions into ‘options’ or subroutines (Botvinick, Niv, & Barto, 2009).

Algorithms derived from these assumptions (see Woergoetter & Porr, 2007 for a brief overview) are effective when applied to certain practical tasks and are also helpful in elucidating some of the brain underpinnings of RL (to be discussed in Section 4). Both the model-free and model-based approaches do, however, have conceptual problems that stem from the very assumptions that help define them.

On the one hand, because ‘a policy [action] corresponds to a stimulus-response (S-R) rule of animal learning theory’ (Barto, 2013, p. 21), model-free RL, which maps inputs directly to actions, inherits the problematicity of the S/R assumption, which has been discussed in Section 2. This observation does not detract from the demonstrated effectiveness of classical model-free RL algorithms: the successes of RL do mean that learning S/R associations can be done well (and even optimally; cf. Pack Kaelbling, Littman, & Moore, 1996, p. 240), but they do not make S/R association better suited to explaining or modelling behaviour in general. Model-based RL, on the other hand, is much more powerful in that it aims to construct a potentially sophisticated representation of the environment and to use that to reason about possible actions and their consequences. This power has remained, however, largely unrealised, for several reasons.

3.1.1. The problems with assuming exclusively external reward

The supervised learning algorithms used with the DNs (recall Section 2.2) adjust the network's parameters so as to minimise the total error on a training data-set. The error being defined as a measure of discrepancy between the desired and actual responses to a stimulus, this approach to S/R learning is clearly knowledge-intensive: it requires an external teacher that knows the correct response to every conceivable stimulus. In contrast, RL algorithms are driven merely by the magnitude (or even just the sign) of the reward that the environment, acting as the teacher, apportions to the learner's behaviour.

The difference between error minimisation in DNs and cumulative reward maximisation in RL is very important: ‘Although supervised learning clearly has its place, the RL framework's emphasis on incessant activity over equilibrium seeking makes it essential – in the author's opinion – for producing growth-oriented systems with open-ended capacities’ (Barto, 2013, p. 34). The pursuit of external reward by the common RL algorithms is, however, problematic in several respects.

First, although most laboratory studies of RL in human and other animal subjects involve the delivery of a reward at frequent intervals (in some studies, after each trial), in realistic animal behaviour actual environmental reward is rare (think of lions or cheetahs, which may go for days without making a kill). Such sparse rewards are rarely informative enough to allow effective credit assignment (apportioning credit or blame to past actions; cf. Section 2). Moreover, many types of behaviour, such as exploration (as opposed to exploitation; Dayan, 2013)7 or play (Graham & Burghardt, 2010), do not by definition carry external reward, yet are important and even crucial for the animal's development and adult functioning.

In ethology, if not in robotics and AI, the ultimate theoretical concern with regard to the sparsity of the external reward has to do with evolutionary viability. On the one hand, much of RL research assumes without questioning that agent behaviour is driven (or, normatively, must be driven) by the pursuit of occasionally rewarding outcomes for its actions. On the other hand, the only outcome that really counts is the final one: the agent's number of offspring.

In one respect, however, the sparsity of external reward in natural ecosystems may be a blessing. If let loose, RL driven by external reward pushes behaviours to extremes of stereotypy, resulting in the emergence of habits and ultimately addiction (Graybiel, 2008) and the loss of agency. This suggests that the agent needs to balance the learning of habits (which serve as useful behavioural shortcuts) with preserving agency, so as to promote actions directed at goals that are in some sense the agent's own, rather than the reinforcer's.

3.1.2. The problems with assuming fixed state space and utility

Behaviour by definition unfolds over time and must be managed so by the situated agent, which responds in time to ‘slings and arrows of outrageous fortune’ and initiates and directs the pursuit of its own goals. As the agent executes actions and accrues experience, its state space, which represents the combined system comprising the agent and its environment, may grow. If this growth involves radical novelty, as in the emergence of new dimensions or the addition of new states, a closed-form treatment of the resulting dynamically changing state × utility landscape becomes impossible (cf. Kauffman & Clayton, 2006).

The open-endedness of real-world situations renders DN-style supervised learning (e.g. minimising an objective function defined as regularised empirical loss over a training set) ineffective. The difficulty of formulating an objective function that would anticipate the needs of real-time control of behaviour may be intuitively appreciated by considering the infinite regress that ensues if, for instance, an agent attempts to learn a generative model of a certain dynamic aspect of the world. At a minimum, this calls for approximating the joint probability over observable measurement and control variables, possibly along with some posited hidden variables (Körding & Wolpert, 2006).

It may seem that in a RL setting, this task might well be undertaken by a DN, with the reward schedule added as another, key dimension to the joint probability distribution. Indeed, a system equipped with a convolutional DN for acquiring the RL action-value function recently achieved human-level performance in learning to play dozens of computer games directly from raw image and running score data (Mnih et al., 2015). According to the system's designers, this impressive feat was made possible by ‘harnessing state-of-the-art machine learning techniques with biologically inspired mechanisms’, namely, experience replay (Davidson, Kloosterman, & Wilson, 2009) and a sparse (as opposed to continuous) updating of the action values, corresponding to infrequent reward.8 For a realistic motivated agent, however, the utility of actions is not only subjective, but may depend in complex ways on the agent's entire prior history of interaction with the world, including other agents. Thus, in model-based RL, according to the rational perspective, ‘the model which is being learned might be influenced by all and any knowledge that the agent has’ (Chater, 2009, p. 352). Asymptotic convergence of RL (let alone DN) learning in such situations may be too much to expect.9

3.1.3. The problems with assuming fixed association of states and actions

Even if an agent using RL were to track successfully the changes in the environment and in its own utility function, the assumption that there is a truth of the matter to the question of what to do next – that the mapping from states to actions is fixed for each epoch (albeit initially unknown and possibly changing over time) is overly limiting. In the terminology of Section 2.4, the behaviour of an agent that operates under this assumption is ‘robotic’. Animal behaviour, in contrast, is often flexible. Chater (2009) recounts the extensive evidence from empirical studies, indicating that RL ‘is influenced by a wide range of “cognitive” factors, which are difficult to explain using a mechanistic perspective on reinforcement’.10

In particular, when the animal's learning objective, as established in a series of trials, is suddenly devalued, or has its valency switched, the learner adjusts its behaviour immediately rather than on the usual RL schedule – a finding that Chater (2009, p. 357) describes as ‘perhaps the most clear-cut case of an apparent contrast between a “reason-based” cognitive viewpoint and putative RL mechanisms’. Thus, gradual modifications posited by RL conflict with observed behaviour, which is ‘to a good approximation, all-or-none […], as if the animal is adopting or rejecting a hypothesis about a putative connection in the environment. Smooth learning curves emerge only from data aggregated across many animals’ (Chater, 2009). Likewise, in their recent review of the brain basis of RL, Haber and Behrens (2014) describe such shifts as stemming from ‘clashes between reasons’, noting that the cognitive complexity of behaviour may account for the intricate circuitry that supports it (cf. Section 4.3).

3.2. A broader view of RL: intrinsically motivated learning of the environment

By relaxing the assumptions built into the varieties of RL discussed so far, we can formulate a broader view of learning as it applies to realistic behaviour. The broader formulation of RL should aim to balance reward-driven acquisition of habits with endogenous agency and motivation, should accommodate a changing, open-ended environment and should allow for the growth of the agent's knowledge of it and for the fluidity of the agent's patterns of utility, which determines the meaning of reward. As in the beginning of Section 3, two aspects of the proposed formulation are discussed under separate headings: what to learn, and how to do it.

3.2.1. Learning the world

Briefly, flexible behaviour requires that agents learn as much as possible about the physical and the causal structure of their environment. Ideally, an agent would use this knowledge on the fly to reason about and select among the available actions, or to devise novel actions, given its goals and its best estimate of the current state of the world. With regard to the physical layout of the environment, and in the context of an extremely simple foraging task, this strategy was shown by Kolodny et al. (2014) to be evolutionarily advantageous over several others, including backward chaining of the kind used by some RL algorithms.

More generally, just like learning the layout of a physical space can support smart navigation behaviour such as shortcut-taking (Tolman, 1948), so learning the layout of an abstract problem space can support effective and creative problem-solving (see Edelman, 2008, ch. 8 for an overview). Finally, the learner may aim to acquire causal dependencies – a feat of which many animal species appear to be capable (Redgrave et al., 2010). Causal learning is particularly important for goal-directed as opposed to habitual control, because representations that make causal dependencies explicit afford informed intervention (Chater & Oaksford, 2013; Holyoak & Cheng, 2011; Pearl, 2009). A further discussion of these matters is beyond the scope of this paper.

3.2.2. Intrinsically motivated learning

The idea that an animal would do well to learn the structure of its environment has been around for a long time – and with it the question of the source of its motivation for doing so. According to Barto (2013, p. 29),

Tolman's (1932) theory of latent learning, for example, postulated that animals are essentially always learning cognitive maps that incorporate confirmed expectancies about their environments. But according to Tolman, this learning is unmotivated: it does not depend on reinforcing stimuli or motivational state. This is different from Berlyne's view that surprise engages an animal's motivational systems. Therefore, it is necessary to distinguish between learning from surprise as it appears in supervised learning algorithms and the idea that surprise engages motivational systems.

An algorithmic treatment of surprise and its use in motivating exploration and acquisition of world knowledge can be found, for instance, in the work of Schmidhuber (2009b), who ties surprise to the ability to compress data and to predict novel inputs (cf. Luciw, Kompella, Kazerounian, & Schmidhuber, 2013; Wu & Miao, 2013). A discussion of the differences between the prediction-based approach and that derived from a computational formulation of competence can be found in the introductory chapter by Barto (2013, p. 31) to a recent volume dedicated entirely to intrinsic motivation in RL (Baldassarre & Mirolli, 2013).

A particularly interesting question is whether or not the capacity for purely internal motivation (as opposed to external reward) is evolvable. In a recent evolutionary simulation study, Singh et al. (2010) demonstrated that internal motivation indeed emerges in a population of foraging agents under the right conditions. In two experiments, they showed (i) that ‘evolutionary pressure to optimise fitness captured in the optimal reward framework can lead to the emergence of reward functions that assign positive primary reward to activities that are not directly associated with fitness’ and (ii) that ‘the agent's internal environment […] provides an inverse recency feature’. This, in turn, makes it possible for the agent to single out state-action pairs that it has not recently experienced, enabling exploration.

The key phrase in the preceding paragraph, and possibly in the entire report it quotes from, is internal environment. Reviewing information-seeking and curiosity, Gottlieb et al. (2013, p. 3) remark that ‘the sources of all of an animal's reward signals are internal to the animal.’ Barto (2013, fig. 2), likewise, states that ‘the environment is divided into an internal and external environment, with all reward signals coming from the former’. This observation is true in the simple and immediate sense that the value of any reward is in the eyes of its recipient: water or food are rewarding to me only insofar as I am thirsty or hungry – a truism that is known in economics as the paradox of value and that has been discussed at least since the time of Adam Smith.

More importantly, however, the maxim is also true in a structured sense: to be of use, my internal environment must be sufficiently complex to reflect the layout of the relevant state space – physical layout in case of spatial exploration; abstract topology in case of abstract problem solving. In particular, in the example of Gottlieb et al. (2013), it takes a structured internal environment to support the estimation of recency with respect to states. In other words, our best guess as to what behaving animals do (and what behaving artificial agents should aim to do) is learn to represent their world.

In concluding this section, it is worth noting that the idea that to learn means ultimately to learn to represent the world is as yet by no means a matter of consensus. Of all the chapters in Baldassarre and Mirolli (2013), only one mentions map-learning: ‘Scott and Markovitch (1989) presented a curiosity-driven search method by which an agent's attention was continually directed to the area of highest uncertainty in the current representation, thus steadily reducing its uncertainty about the whole of the experience space’ (Barto, 2013).

4. Architecture

With the broader picture in mind of behavioural tasks and of the learning needs that arise from them, we can now have a closer look at some of the popular architectural assumptions that are being made in the cognitive modelling literature. In this section, I discuss two sets of such assumptions, one associated with the DN architecture and the other with RL models.

4.1. The cortex: multiple hierarchies

Practitioners of machine learning often motivate their use of a layered, uniformly connected architecture, such as that of DNs, by claiming a parallel with the hierarchical architecture of the brain.11 The scope of this analogy is, however, extremely narrow. It is true that a DN may succeed in visual recognition and categorisation tasks insofar as its structure mimics the architecture of the visual cortex (Poggio, Mutch, Leibo, Rosasco, & Tacchetti, 2012; Yamins et al., 2014). The visual cortex is, however, only one of many areas and circuits that comprise the mammalian brain (for a particularly forceful illustration of this fact, see Cotterill, 2001, fig, 7) and is entirely absent in other vertebrate species (e.g. birds) that are, generally speaking, as good at vision as mammals. In any case, the DN–brain analogy is strained even if we limit the consideration to the mammalian cortex.

While hierarchical in some respects (and perhaps surprisingly deep; Hilgetag, O'Neill, & Young, 1996), the architecture of mammalian isocortex is generally much more complex than assumed by the standard DN–brain analogy. To begin with, there is a major anatomical and functional division between posterior and frontal cortex (Fuster, 2008; Stuss & Knight, 2002). Functionally, this division is thought to correspond to a trade-off between, on the one hand, statistical generalisation and semantic association, supported by overlapping distributed representations in the posterior cortex, and, on the other hand, working memory and executive control in the frontal cortex (Atallah, Frank, & O'Reilly, 2004). As described next, the posterior and the frontal parts of the cortex have each their own hierarchies (Fuster, 2008, p. 339).

The posterior cortex contains three anatomical and functional hierarchies – visual, auditory and somatosensory/motor – whose common apex is situated at the hippocampus, an evolutionarily older allocortical structure that is part of the medial temporal lobe (Merker, 2004). The division of labour between posterior cortex and the hippocampus illustrates another memory task-based trade-off. As just noted, the hierarchical converging/diverging connection pattern in the posterior cortex is used for integration of lifetime memory. At the same time, the CA1–CA3 circuit in the hippocampus, which is wired like a crossbar switch between ascending and descending posterior pathways and has recurrent connections, supports episodic memory in the form of sparse representations of current stimuli, which it places in the context of long-term statistical patterns distributed across the entire posterior cortex (Atallah et al., 2004; Merker, 2004).

The frontal cortex contains distinct ‘stripes’ or columns of neurons capable of sustained firing, which support active short-term maintenance of information or working memory (Atallah et al., 2004). The level of processing depends hierarchically on cortical location, with the more rostral regions corresponding to progressively more abstract tasks (Badre, 2008; Botvinick, 2008). The dynamics of task control across the stripes is, however, neither simply convergent/divergent (as it would be in a purely pyramid-like architecture), nor, indeed, confined to the isocortex.

4.2. Beyond the cortex

Rather than being a largely cortical process, the dynamics of task control by the frontal lobe involves multiple parallel loops that connect frontal and posterior cortical regions with the hippocampus, thalamus, cerebellum and the basal ganglia, a collection of subcortical nuclei long implicated in RL (Graybiel, 2008; Redgrave, Vautrelle, & Reynolds, 2011). Massive involvement of subcortical structures is equally prominent in the posterior cortex. In the visual pathway, in particular, the ascending retinal connections in the feedforward hierarchy are dwarfed by other influences: in the lateral geniculate nucleus of the thalamus, only 5–10% of the input comes from the retina, the rest descending from layer 6 of various visual cortical areas and ascending from the brainstem (Sherman & Guillery, 2002). Another thalamic nucleus, the pulvinar, is plugged into both posterior and frontal cortical hierarchies at every stage, so that ‘often, and perhaps always, when cortical areas have a direct connection, they also have a parallel one through thalamus’ (Sherman, 2012). Indeed, the manner and degree of involvement of the thalamus in cortical processing is such that ‘the distinction between so-called “sensory” and “motor” areas of cortex is of limited use’ (Sherman, 2012).

We can approach the anatomical complexity of the mammalian brain by trying to interpret it in the context of real behaviour and real learning tasks. Taking as an example foraging, which came up earlier in this paper, one may observe that its brain mechanisms include both cortical and subcortical structures. In primates, in particular, the newly evolved frontal cortical areas have been hypothesised to

reduce foraging errors by generating goals from current contexts and learning to do so rapidly, sometimes based on single events. Among the contexts used to generate these goals, the posterior parietal cortex provides the new prefrontal areas with information about relational metrics such as order, number, duration, length, distance and proportion, which play a crucial role in foraging choices (Genovesio, Wise, & Passingham, 2014).

The circuits that make this coordination and control possible include, notably, the thalamus, with its role in attention and in active vision (Wurtz, McAlonan, Cavanaugh, & Berman, 2011), and the basal ganglia, which help set priorities for attentional shifts (Krauzlis, Bollimunta, Arcizet, & Wang, 2014) and play a crucial role in RL.

4.3. The basal ganglia: far-reaching, intricate circuits

Many computational models of learning and memory have aimed at interpreting the gross neuroanatomy of the basal ganglia in terms of reward- and prediction-related switching of the so-called direct and indirect pathways that channel the corticostriatal circuits (Botvinick et al., 2009; Cohen & Frank, 2009; Niv, 2009; O'Reilly, Noelle, Braver, & Cohen, 2002; O'Reilly & Frank, 2006; Redgrave et al., 2011). The successes of these models are particularly impressive against the backdrop of the neuroanatomical and functional complexity that remains to be explained.

The main functional challenges are, as already mentioned, to maintain a balance between habit-learning and supporting agentic behaviour, as well as to provide a basis for learning the environment in the form of a structured internal motivation signal. It has been suggested that this is done by distinct circuits passing through the basal ganglia insofar as they instantiate two mechanisms of RL: one driven by internal motives (manifesting agency) and the other by external motives (responding to reward) (Hikosaka & Isoda, 2010; Redgrave et al., 2010, 2013).

With regard to the neuroanatomy, the challenge arises to a large extent from the sheer number of brain areas and structures that seem to be involved in these processes – an extent that is by rights expected from a general framework that encompasses all of behaviour. In this sense, the surging interest in RL has already yielded conceptual dividends for the neural computation community by encouraging theorists to look beyond areas that have long been studied in isolation, such as the sensory cortex or the hippocampus. In addition to their far reach in the brain, however, the circuits in question exhibit also some striking anatomical characteristics.

First, they contain multiple loops running in parallel between distinct corresponding subparts of the areas involved, from the prefrontal cortex (PFC), via the striatum and the thalamus, and back to PFC (Badre, 2008; Botvinick, 2008; Graybiel, 2008). The corticostriatal projection is global in the sense that the entire isocortex is mapped onto the surface of the striatum, in a manner that largely preserves the topography of the originating cortical areas (Redgrave et al., 2011). Furthermore, the loops that pass through PFC are organised along the rostro-caudal dimension into distinct stripes (Badre, 2008; Fuster, 2008), which I mentioned earlier.

Second, along the way the loop circuits are modulated by selective inputs from other parts of the brain (Redgrave et al., 2011), forming a pattern of connections that is much more complex than the familiar direct/indirect pathway segregation.12 For instance, the ventral striatum receives topographically distinct projections not only from isocortical areas in the frontal lobe (in order, from the ventral pole of the striatum, these are: ventromedial, orbitofrontal, anterior cingulate, dorsal prefrontal and premotor cortices), but also from allocortical and periallocortical areas (hippocampus, insula), as well as from the amygdala, ventral tegmental area and substantia nigra (Haber & Behrens, 2014. fig. 8). These projections form not a hierarchy but a loose, loopy network (Thompson & Swanson, 2010) of highly complex graded associations whose details are still being uncovered (Haber & Behrens, 2014).

Third, there are clear and precise internal divisions within the areas connected by the circuits. Just in the striatum, examples include the above-mentioned topography of the projections into the ventral part, the ventral/dorsal gradient (Bornstein & Daw, 2011), and the cytoarchitectonic and functional distinctions between striosomes and matrisomes (Amemori, Gibb, & Graybiel, 2011).

Is such an elaborate architecture essential for supporting complex sequential behaviour? From the evolutionary standpoint, the answer to this question must be affirmative. Evolution is clearly capable of putting in place highly uniform neural circuits (for instance, in the cerebellar cortex, the sheaf of parallel fibres that run at a right angle to the flat dendritic trees of Purkinje cells, or, in the hippocampus, the crossbar switch implemented by the CA1–CA3 loop, which I mentioned earlier), so what looks like vaunting complexity of some brain circuits is likely there for a reason.13

4.4. A lesson from the mammalian brain: circuitry matters

The intricate anatomical patterns and distinctions discussed above – which constitute as good an argument as any for the opening sentence of Atallah et al. (2004, p. 253): ‘The brain is not a homogeneous organ’ – are being increasingly used to motivate relatively elaborate functional interpretations. For instance, the striosome/matrisome circuitry has been described functionally in terms of a gated mixture of experts (Amemori et al., 2011). Similarly, the ventral basal ganglia (‘the crossroads of space, strategy, and reward’; Humphries & Prescott, 2010) are thought to support replay of episodic memory, while tracking reward and implementing action selection by controlling the relative contribution to the current action of the different frontal cortex stripes (Jin, Tecuapetla, & Costa, 2014).

The elaborate circuitry and functional anatomy of real brains stands in contrast to the common DN architecture, which is uniform (in that all units in a given layer usually have the same pattern of connections) and distributed (in the usual representational sense). A long-standing concern about modelling the brain with neural networks has been the opaqueness of the representations they learn. With their especially large number of degrees of freedom, DNs are particularly susceptible to this concern: a recent paper by Szegedy et al. (2014) documented and analysed widespread hypersensitivity to perturbations of representations learned by DNs. For these reasons, distributed representations, which characterise DNs, are difficult to put to use in inference (reasoning) and control tasks.

To understand the source of this difficulty, consider again the concept of credit assignment. In any network- or graph-structured representation, it requires that both learning and use cause information to be propagated through the graph. The more complex that representation, the more difficult both temporal and structural aspects of credit assignment become. Using representations that are highly distributed over not just one but many hidden layers (as in DNs), as well as over time (as when serial behaviour needs to be controlled) greatly complicates credit assignment.

While the latest DN algorithms alleviate credit assignment during training, the ‘reuse’ over time of the hidden units remains a problem, whose severity increases as the temporal extent of the behaviour grows. Although attempts are often made to model hierarchically structured sequential behaviour such as language learning and production in shallow uniform distributed architectures (e.g. in Simple Recurrent Networks; Chang, 2002; Elman, 1990, 2009; or Echo State Networks; Tong, Bickett, Christiansen, & Cottrell, 2007), such attempts typically only work on toy data or on tasks of limited behavioural relevance, such as learning small artificial grammars or predicting the part of speech of the next word in a sentence. Recurrent varieties of DN, which may also combine deep representations with RL, typically do better, mainly because they do a much better job of credit assignment (Schmidhuber, 2015). However, as noted in Section 2, they are limited by their treatment of every problem as an exercise in S/R association – as in mapping entire sentences from one language to another (Sutskever et al., 2014), or as in language modelling, which amounts to assigning probabilities to strings of words (Goodman, 2001; cf. Schmidhuber, 2015, sec 5.20).14

One way to alleviate the uniform recurrent networks’ problem with extended sequential behaviours is to ‘unfold’ the recurrent architecture, letting space stand for time, so that longer sequences are represented by longer chains of units connected in series (an approach that becomes more attractive if used in conjunction with hierarchical chunking, as suggested already by Lashley (1951); cf. the notion of hierarchical RL with ‘options’ representing chunks of elementary actions (Botvinick et al., 2009)).

This implementation-level design choice is supported by a behavioural-level observation: complex sustained sequential behaviour – migration, foraging, courtship rituals, glade skiing, birdsong, language – amounts to navigating through a shifting labyrinth of discrete choices. Every critical point in such a sequence (as in deciding whether to pass to the left or to the right of the next tree down the glade) is a causal nexus, at which one and only one choice must be made; a superposition of alternatives will not do. Thus, it seems natural to insist that the control of such behaviour be serially local, in the sense of relying, for each choice, only on representations that are available locally at the unit that directs the sequence, even if the representations that guide the choice are themselves distributed.15 Furthermore, if the sequences comprising the behaviour need to be learned (as in birdsong or language, but not in zero-day glade skiing), localist representations of critical control junctures have the extra benefit of easier credit assignment. A design principle for a network architecture based on the above ideas is outlined in the next section.

5. Structured connectionism redux

A straightforward way of making the topology of the network (and not only its connection weights) matter in dealing with sequential behaviour is to have the learner construct incrementally a graph-like localist representation of its sensorimotor experience. I shall illustrate this approach on the problem of learning and using natural language.

Given a sufficiently powerful mechanism for S/R learning, such as DNs, one may try to reduce language acquisition to the minimisation of an appropriately formulated objective function. In computational linguistics, a related idea has been endorsed by the proponents of the ‘new Empiricism’ (Goldsmith, 2007) (including an earlier version of the present author). According to this approach, learning proceeds by estimating (perhaps dynamically, using RL), from observed corpus data, a probabilistic language model, which should ideally capture the joint probability distribution over word sequences and context, the latter including both more language (discourse history) and extra-linguistic materials, such as the speaker's motivation, semantic intent and so on. The resulting model can then be used to generate utterances by following the most probable paths through the space of possible word sequences.

We saw, however, that such learning does not seem to be possible in a uniform distributed architecture. In particular, in such an architecture, in which part–whole relationships over sequences are not made explicit, observations of entire utterances may not readily support the acquisition of the representations of their parts. Crucially, language users are typically interested in saying not what the world expects to hear from them, but rather what they intend and mean, and doing so – exhibiting true, agentic productivity – is impossible without being able to manipulate parts of utterances.

This suggests that the learner of language should aim at constructing a generative model that explicitly represent the graph structure of the utterance space, as suggested in the closing paragraphs of Section 4.4. In such a model, language (and other types of complex sequential behaviour, such as spatial exploration; Chrastil, Warren, & Sutherland, 2014; Muller, Stead, & Pach, 1996) can be generated by injecting activation into a node in the representation graph and letting it spread, as illustrated in Figure 1 for the case of language learning and its use in machine translation (Edelman & Solan, 2009). As already noted, the S/R mapping needs of such models, such as recognising when the conditions for activating a localised-representation node are met, can be fulfilled by hierarchical distributed-representation DNs.

Figure 1 Some of the representational principles behind a structured-connectionist model of language acquisition (Solan et al., 2005), and of its use in English to Spanish translation (Edelman & Solan, 2009). The source-language grammar, learned from a corpus in an unsupervised fashion, is a directed graph over terminals (such as cat or the), collocation patterns (which may be only partially lexicalised, as in ‘the _–- is running’), and equivalence classes (such as the set {cat, dog}). The terminals evoked by the novel input (the sentence ‘The cat is running’) are shown in bold. These in turn activate corresponding target-language terminals on the right; patterns and equivalence classes also activate their counterparts. Due to polysemy (here, between the different senses of running) and initial structural ambiguity, a source element can activate more than one target element; the disambiguation is enforced by the context (Karov & Edelman, 1998) via the interaction between the activated target elements and the target language model, which, along with the target-language grammar, is also learned automatically. Given a set of active elements, the target-language model constructs the most probable sentence that is consistent with the source-language original.

The conceptual roots of this approach, which may be called structured connectionism, include the old-style connectionism (Feldman & Ballard, 1982; Feldman, Fanty, Goddard, & Lynne, 1988) and the augmented transition network (ATN) methods in natural language processing (Woods, 1970). Structured connectionist networks can directly represent sequential order and can support functional abstraction, as indicated, for instance, by the newer work on embodied construction grammar (Chang, Feldman, & Narayanan, 2004), where units stand for concept frames such as ‘walk’ and directed connections represent the sequential ordering of units.

A structured connectionist network has been the representation of choice of the ADIOS model of unsupervised language acquisition and production (Solan, Horn, Ruppin, & Edelman, 2005), which underlies the example of Figure 1, and its successor, U-MILA (unsupervised memory-based incremental language acquisition; Kolodny, Lotem, & Edelman, 2015). The graph vertices in these models initially represent words (or other symbolic primitives, such as phonemes); edges initially stand for actual observed transitions. With learning, recursive abstraction leads to the emergence of hierarchically structured phrase units and their incorporation into the graph.

Such graph formalisms can be made computationally quite powerful. ATNs, in particular, are equivalent in that respect to Turing machines. Another powerful graph-based representational formalism for structured connectionism is higraphs, introduced by Harel (1988). Intuitively, a higraph is a Venn diagram with a unique blob for every set/state, hyper-edges connecting states and optional composite Cartesian-product states. Importantly, higraphs have been given procedural semantics, in the form of statecharts.16 A recently developed graphical computational formalism related to statecharts, Live Sequence Charts (Damm & Harel, 2001; Harel & Marelly, 2003), may be particularly well-suited for modelling complex sequential behaviour, as conjectured by Kolodny and Edelman (2015).

Informally, it seems that the higraph-like representations offer the right primitives for modelling, on one level, complex sequential behaviours, and on another – brain circuits and processes.17 In particular, the ability of a higraph grammar of behaviour to be in a superposition of states may model the possibility of a brain network having several foci of activity at the same time. The higraph's combination of nested structure with sequential structure may help model certain brain circuits. Finally, higraphs, as any other structured connectionist architecture, allow meaningful activation to be injected into nodes in a targeted manner – something that is difficult, if not impossible, to do in a distributed representation.

6. Conclusions

At the end of the passage on the ‘circuit’ of experience and behaviour quoted in Section 2.3, Dewey (1896) observes that in psychology and brain science the evolved form is indicative of function: ‘The end has got thoroughly organised into the means.’ The idea that understanding behaviour goes hand in hand with understanding how it relates to the fine details of brain circuitry sits well with the less corticocentric among the current theories of how the brain/mind works – e.g. theories of language that consider the entire cortical–BG circuit (Lieberman, 2002; Ullman, 2001) and theories of birdsong that tie it to theories of language (Bolhuis, Okanoya, & Scharff, 2010).

Predictions of such theories are being steadily corroborated. For instance, with regard to the posited role of the basal ganglia in language acquisition and use, an impairment of RL in general appears to accompany developmental language impairment (Lee & Tomblin, 2012).18 Likewise, with regard to the posited involvement of the hippocampus in language acquisition (Edelman, 2011), individual differences in hippocampal neuroanatomy in infancy predict language performance a few months later (Can, Richards, & Kuhl, 2013). Still, a single overarching framework for integrating such findings with a comprehensive understanding of the brain circuitry – which, as I suggested in Section 4.4, is elaborate, as opposed to uniform – is lacking.

In this paper, I used problem- and representation-level analysis and some neuroanatomical data to argue that the popular DN are in principle unsuited for modelling complex sequential behaviour in animals, or of supporting it in artificial systems, and that RL methods could do so better if extended so as to rely more heavily on intrinsically motivated exploration. These ideas underlie the modest constructive proposal outlined in the preceding section, which offers a complementary, structured connectionist approach to general problem of complex sequential behaviour. Most of the work in developing this proposal remains to be done. This work includes the following:

• Making the circuitry of structured connectionist models such as U-MILA (Kolodny et al., 2015) more brain-like. This means working out not just an area-level flowchart but also a detailed wiring diagram for specific cell types (as in Wickens, 2009, fig. 1) across the areas connected by each proposed circuit.

• Making the models more realistic psychologically. The U-MILA model, for instance, lacks a social reinforcement mechanism, which is clearly at work in infant learning (Goldstein et al., 2010).

• Developing and testing predictions specific to the models. One possible focus for these could be based on the limitations imposed by model architecture and functioning on generalisation and variable binding (cf. Kriete, Noelle, Cohen, & O'Reilly, 2013).

• Developing better algorithms for training the models. With regard to language learning, existing algorithms for statechart synthesis (Harel, Kugler, & Pnueli, 2005) work from abstract scenario-based requirements rather than sample outputs (which is what is available in a language acquisition setting). At the same time, state-of-the-art models in graph-based language learning (Chang, 2008; Kolodny et al., 2015) employ heuristics, Bayesian or other, in deciding when and how to modify the grammar in the face of new experience. This task may be made easier by resorting to the formal Bayesian methods of Tenenbaum, Kemp, Griffiths, and Goodman (2011).

The structure of natural languages, with their elaborate syntax, is probably the ultimate challenge for any theory of complex sequential behaviour. The ultimate test of a candidate understanding of the design of the brain and a corresponding computational model of human behaviour would, therefore, be in replicating not just foraging, or tool use, or the dynamics of social interaction, but also the range of syntactic phenomena and semantic inference that are currently beyond the reach of natural language engineering.

Acknowledgements

I have benefited from discussing the ideas behind this paper with Björn Merker, Barb Finlay, Oren Kolodny, Arnon Lotem, David Harel, Assaf Marron, Hiroyuki Nakahara, Kazuo Okanoya, Okihide Hikosaka and the participants of the Shonan Workshop on Deep Learning, May 2014. I thank the anonymous reviewers for their comments and suggestions.

Disclosure statement

No potential conflict of interest was reported by the author.

Notes

^1. One of the names of this type of bias, which is proverbial in English, is the Law of the Instrument or Maslow's Hammer: ‘I suppose it is tempting, if the only tool you have is a hammer, to treat everything as if it were a nail’ (Maslow, 1966, p. 15).

^2. Due to space limitations, each section presents bare arguments and cites only a few select sources. No attempt is made to cover all behaviour or the entire brain.

^3. I call a layered network uniform if all units in a given layer have the same pattern of connections (with possibly different weights); cf. Section 4.4. Input-output learning is supervised when driven by an error signal that explicitly refers to the desired output for each input training data point; unsupervised learning relies on some other measure of goodness of the acquired model, such as the input reconstruction error in autoencode r architectures (DeMers & Cottrell, 1993) or an information-theoretic measure in MDL- or compression-based approaches (Adriaans & Vitányi, 2007; Grunwald, 2005; Schmidhuber, 2009a). A probabilistic model is generative if it captures the joint probability distribution over the variables of interest; a discriminative model, in contrast, aims to capture the conditional probabilities of class labels for the outputs, given the inputs.

^4. This insight has been vindicated by the discovery in the human brain of the default network activity – a complex dynamics that serves as the foundation for, and is modulated by, exogenous stimuli and endogenous goal-oriented thinking (Raichle, 2006, 2010).

^5. Cf. Konishi (1971, p. 60): ‘Ethologists found the explanation of behavioral development in terms of reflex unsatisfactory. They held a strong antireflex view from the beginning.’

^6. Cf. ‘All organisms with complex nervous systems are faced with the moment-by-moment question that is posed by life: what shall I do next?’ (Savage-Rumbaugh & Lewin, 1994, p. 255).

^7. Any good foraging strategy includes both exploitation of resources and exploration of the environment (Kamil & Sargent, 1981; Niv, Joel, Meilijson, & Ruppin, 2002).

^8. Even so, the system's performance remained far below that of human players on games, such as Montezuma's Revenge, that involve complex, hierarchically structured environments.

^9. A more formal approach to this issue may be based on the so-called Good Regulator theorem of Conant and Ashby (1970), who proved that, under certain probabilistic assumptions, ‘every good regulator of a system must be a model of that system’. Interestingly, the key assumption on which the proof is based is that the overarching objective in control is homeostasis (not surprising, given Ashby's fame as the inventor of the concept of homeostat (Ashby, 1952), which by definition does not leave room for endogenous choice on the part of the agent (the ‘regulator’). Cf. (Barto, 2013, p. 32), citing Klopf: ‘homeostasis should not be considered the primary goal of behavior and learning and that it is not a suitable organizing principle for developing artificial intelligence.’ An intriguing conjecture (which I offer here without proof) is that allowing for endogenous choice would lead to infinite regress, severely limiting the applicability of the Good Regulator theorem to realistic animal behaviour.

^10. For an evolutionary angle on the question of behavioural flexibility, see Dennett's (1995) notion of ‘The Tower of Generate and Test’ and related ideas, which draw on the classical work of Lorenz (1977).

^11. For example:

Whereas most current learning algorithms correspond to shallow architectures (1, 2 or 3 levels), the mammal [sic] brain is organized in a deep architecture with a given input percept represented at multiple levels of abstraction, each level corresponding to a different area of cortex. (Bengio, 2009)

^12. On one popular account of the basal ganglia function, the action control circuit consists of two pathways – the direct one, which exclusively selects the target action (GO), and the indirect one, which suppresses all other actions (NO–GO). While this idea has led to much excellent modelling work (e.g. Atallah et al., 2004; Cohen & Frank, 2009; O'Reilly & Frank, 2006), it oversimplifies not only the anatomy of the circuit (cf. Redgrave et al., 2010, fig. 1), but also its functioning: as shown by Jin et al. (2014), the appropriate action is selected through joint, graded and distributed activity of the direct and indirect pathways.

^13. The reason need not be statable in a simple closed form: the complexity may be due to an historical accident such as an accumulation over evolutionary time of ‘strata’ of circuits in a subsumption architecture (Brooks, 1986). For an overview of brain evolution, and in particular of the vertebrate brain plan, see Striedter (2005).

^14. This remark holds also for the DN-based language model of Collobert and Weston (2008), who used it to map sentences to part-of-speech tags, chunks, named entity tags, semantic roles, semantically similar words and the likelihood that the sentence makes sense (grammatically and semantically); it also holds for the DN-based parser of Hermann and Blunsom (2013), whose tasks were sentiment estimation and phrase similarity ranking.

^15. For an outline of distributed control over discrete pathway switching, see Miller and Cohen (2001) and Botvinick et al. (2009, fig. 6).

^16. Statecharts is an industrial-strength software specification tool and a predecessor of the UML. See Harel (2007) for a historical perspective and Mehlmann and André (2012) for a recent application of related ideas to multimodal parallel processing for user interaction.

^17. Higraphs and related formalisms may thus constitute precisely the kind of tool that computational ethologists are calling for: ‘Given these complexities, it is not surprising that a general, computationally sound approach to describing behaviour using conventional descriptors has not yet emerged, since it is unlikely to be manageable “by hand” (as, for instance, ethograms are)’ (Anderson & Perona, 2014, p. 28).

^18. Such findings are consistent with the general idea of a brain-wide reach of circuits involved in complex sequential behaviour. This calls for an investigation of ‘the principles by which different computational elements of reinforcement learning are dynamically coordinated across the entire brain’ (Lee, Seo, & Jung, 2012).