Future steps in visual working memory research

Visual working memory (vWM) is the cognitive function that enables the temporary maintenance of visual information relevant for a current or pending task. For example, when we assemble a piece of flat-pack furniture we often first look at the manual to take in an image of the part we need, after which we look for the same part in the (hopefully complete) package. By definition, vWM is highly flexible, as it can represent spatial, feature, and object information – representations that can be updated, replaced, recombined or forgotten according to task demands. For example, after having assembled one part of furniture, you forget about it and move on to the next step. Thus, vWM is a core component of what makes human cognition so adaptive and flexible in complex environments.

Moreover, vWM research (and related research into visual imagery) provides a window on what we cognition scientists arguably find the most exciting: As purely “mental” representations, somehow activated and manipulated “on line,” visual memoranda provide the epitome of a rich and dynamic internal, cognitive world. As a memory of the present, one could argue that the content of vWM reflects what is currently in the mind's eye, either sampled from the external environment, maintained for later use, imagined, or retrieved from long-term memory.

Finally, in an increasingly more visually driven world, it is important to fully understand our visual capacities, in order to optimally design the visual environment.

Perhaps not surprisingly then, the past two decades have witnessed an extensive growth of vWM studies, as is shown in Figure 1, with currently over 200 publications a year with visual working memory or visual short-term memory mentioned in title or abstract (source: Pubmed.gov, September 1, 2020). These publications contain lively debates about the function, content, control, and capacity of visual working memory, as well as its underlying neural architecture. Research on vWM has clearly crossed the borders between different disciplines, attracting researchers from a wide variety of backgrounds, including perception, attention, consciousness, action and memory. Though perhaps too early to tell, the same graph also suggests that the growth has been slowing over the most recent years, which is a sign of a maturing field, but probably also a field that is at a crossroads, looking for new directions. It is this mix of excitement and the sense of a field at a crossroads that inspired us to compile this special issue, which contains a selection of what may be best described as opinionated reviews and experimental explorations on where the field is and where it is or ought to be going. This selection grew out of a workshop on the same topic held at the Royal Dutch Academy of Sciences (KNAW) in Amsterdam, June 27–28, 2020, in which a superb range of experts on vWM came together.

Figure 1. Number of publications per year as indexed in Pubmed with either “visual working memory” or “visual short-term memory” in the title or abstract. Source: Pubmed.gov (September 1, 2020).

And the contributions do reveal a number of clear trends, as well as a number of central current issues that still await resolution. Here we point these out and, where opportune, briefly add our own take on it.

From the past to the future

The clearest shift in perspective that is currently emerging is from seeing vWM as a memory of towards regarding it as a memory for. The first decade of this century saw an explosion of studies interested in the number and fidelity of items that could be retained, using predominantly delayed match-to-sample or continuous recall tasks (see e.g. Luck & Vogel, 2013, for a review). While this endeavour has generated a wealth of knowledge on what vWM is made of, it left the question what it is made for. The second decade has therefore seen a gradual but undeniable shift towards a focus on the functional use of vWM beyond merely remembering the past (e.g. Myers et al., 2017; Nobre & Stokes, 2019; Olivers et al., 2011). The current special issue reflects this in a number of ways. Still on the sensory side of things, one purpose that vWM is thought to serve is to actively bias attention. Ort and Olivers (2020) focus on the capacity of vWM when vWM content is used to search for multiple potential targets in visual search displays, reviewing the evidence that memory for search may have a different capacity than memory of past visual information. Bocincova et al. (2020) too focus on vWM for the purpose of visual search, but from a modelling perspective, showing that an “off the shelf” model of vWM (Manohar et al., 2019) readily generates biases that mimic attentional guidance effects in visual search. But vWM may also play a central role in biasing attention away from distracting information during encoding, as is argued by Liesefeld et al. (2020). These studies provide examples of how different research fields come together in the study of vWM function.

Memory-for-action

In what is probably the strongest current development in vWM research, a number of reviews further extend the perspective of what vWM is for, by focusing on the relationship to action (van Ede, 2020; Heuer et al., 2020; Van der Stigchel, 2020; see also Olivers & Roelfsema, 2020; Van der Stigchel & Hollingworth, 2018). Specifically, Heuer et al. review evidence that action automatically drives selection of associated information within vWM. For example, programming an eye movement towards a certain location improves memory of an object associated with that location, while programming different hand movements (grasping versus pointing) differentially affects memory for different features, notably orientation and colour. As the authors point out, an important question for future research is to what extent such action-based selection differs from other drivers of selection (i.e. whether action is special), such as standard retrospective cues which are of a sensory nature (Gazzaley & Nobre, 2012; Souza & Oberauer, 2016). Arguably, action-based memory benefits may reflect a two-step mechanism, where first the action goal tells the system which object or location is relevant, after which a sensory-based attention mechanism enhances that information. Conversely, one could argue that even standard retro-cueing involves action selection (Olivers & Roelfsema, 2020). We agree with Heuer et al. that to disentangle these, and a number of other important issues require clever experimenting and probably sophisticated neurophysiological methods.

Additional functional links between vWM and action are discussed by van Ede (2020), who describes evidence how action plays a role in the recruitment of visual memories, how it changes the quality of those memories, and how those memories serve to overcome action-induced changes in sensory input. Moreover, more precise action measures such as microsaccades have recently proven to provide a useful tool for reading out the quality and status of mnemonic representations (de Vries et al., 2018; van Ede et al., 2019; van Loon et al., 2017). In our view, the biggest current challenge that emerges from this is how these bidirectional links between sensory and motor representations are implemented mechanistically (Kruijne et al., 2020).

Van der Stigchel (2020) points out yet another perspective on the role of action in vWM. Action, by definition, enables the cognitive system to interact with its environment, and thus offers the unique opportunity to use that environment for memory purposes. Unlike in the typical laboratory experiment, visual objects in the real world tends to stay quite constant and stable, while we ourselves tend to move around. Rather than precisely and continuously trying to hold on to the visual features of an object, the system could store the goal of moving back to the location of the object to simply sample it again. In other words, actions allow us to use the world as an outside memory. This would also provide vWM with an elegant way around its capacity limitations. While this idea has been around for a good while (cf. O’Regan, 1992), still very little is known about how the brain decides between when to take the effort to recruit working memory and when to leave information in the external world.

Different functional states

Last but not least, different functional purposes either imply different representational states, or different operations applied to the same representational state (causing different representational states further downstream). As Stokes et al. (2020) review, the last decade has seen a good deal of evidence for a functionally ‘active’ state, adopted by memory items used for a current or imminent task, and a functionally ‘latent’ state, adopted by items remembered for a future, prospective task. Either implicitly or explicitly, researchers have associated these states with respectively active and sustained firing versus activity-silent plasticity-based mechanisms. But as Stokes et al., rightly point out, this one-to-one relationship is not necessarily the case. There are multiple ways of recoding or transforming memories in either firing or plasticity patterns. There is thus the risk of confusing questions about the functional role in behaviour with questions about the underlying biological correlate of those representations. We add to this another important unresolved issue, namely how memories are being transformed, on the fly, according to task demands. Which mechanisms can turn functionally active into functionally latent memories, or turn neurophysiologically active into neurophysiological silent memories, and vice versa, at the scale of seconds or less?

One way of changing the representational state of a memory is by adopting a different brain region. This is advocated by Xu (2020, see also Christophel et al., 2018). Xu argues against sensory recruitment as the necessary mechanism of vWM maintenance. According to the sensory recruitment account, vWM makes at least to some extent use of the same representations, and therefore the same brain areas as visual perception. Central evidence for this hypothesis comes from studies showing that vWM content can be reconstructed from visual cortical areas during the delay period (e.g. Harrison & Tong, 2009). However, Xu argues that while visual cortex may be useful, it cannot be necessary for successful vWM maintenance, since objects can also be successfully remembered without visual cortical involvement. The main evidence for this comes from experiments in which irrelevant, distracting stimuli were shown during the memory delay period (Bettencourt & Xu, 2016). This severely disrupted decoding of vWM content from occipital cortex, but not so much performance. Moreover, vWM content could be successfully decoded from posterior parietal cortex during distraction. Xu thus argues that posterior parietal cortex may be the more crucial site for stably storing visual memories for the short term. Although more recent evidence has emerged that vWM content can be successfully decoded from occipital areas even under distracting stimulation (Rademaker et al., 2019), Xu argues that this does not necessarily demonstrate that such occipital areas functionally contribute to the memory.

Interestingly Postle and Yu (2020) argue for what appears to be quite the opposite. They warn that being able to decode the content of vWM from posterior parietal cortex (or frontal cortex for that matter) does not necessarily mean that those areas actually represent vWM content. Rather, they argue, these areas could well be involved in control operations (commensurate the frontoparietal control network's classic role), but that the specific parameters of these operations depend on the stimulus. Postle and Yu then more or less explicitly assume that sensory cortex (they mention occipital and temporal cortex) contains the real vWM content. Although this assumption makes sense, one could argue that Postle and Yu fall in their own trap: If successful decoding of content from frontoparietal networks reflects control mechanisms, the same might then hold for sensory areas. For example, such decoding may reflect where on a stimulus observers attend, covertly, or even overtly (cf. Mostert et al., 2018). In any case, it is clear then that where and how vWM representations are maintained by the brain is far from resolved and still deserves a good deal of further research. A key issue here will be to get a better grasp on what is actually decoded from neural signals using multivariate analyses.

We conclude that the field of vWM research is still as dynamic as the phenomenon it is investigating. Perhaps the take-home message is that our theories should reflect that dynamic nature, as it has become clear that one-dimensional theories on the architecture of vWM will not suffice. In our view, the current special issue clearly identifies the major research questions for the years to come, as well as the directions to take.

Additional information

Funding

This work was supported by the Netherlands Organization for Scientific Research (NWO-Vici grant 453-16-002 to CNLO) and the European Research Council (ERC) (grant agreement No 863732 with SVdS).