Dynamic and flexible transformation and reallocation of visual working memory representations

ABSTRACT

In their recent review, Xu critically assesses the role of early visual areas in VWM storage in the light of new fMRI-decoding studies that seemingly support the sensory storage account. We would like to extend the discussion by highlighting recent findings which suggest that early visual areas can dynamically transform active VWM representations e.g., to activity silent or long-term memory representations. These latent codes evade detection via traditional paradigms as well as decoding methods and hence limit the conclusions that can be drawn about the role of certain brain regions in WM storage. More precisely we claim that a lack or a temporary disappearance of multivariate VWM evidence from early visual brain regions does not imply that these areas are not essentially required to store and maintain active, or currently attended, VWM representations.

KEYWORDS:

• visual working memory
• activity silent working memory
• sensory recruitment
• synaptic theory of working memory
• sensory storage

The role of sensory processing areas in storing visual working memory representations is highly disputed. In a recent publication Xu (2020) critically reviews new findings that aim to support the sensory storage account and concludes that convincing evidence for an essential involvement of early visual areas in VWM storage is still lacking. Instead, they hypothesize that frontal and parietal areas play essential roles in storing VWM representations. Evidence in favour of the sensory storage account comes from different areas. First and foremost, visual processing and visual working memory maintenance elicit similar patterns of neural activity e.g., in early visual area V1 (Albers et al., 2013; Gayet et al., 2017; Harrison & Tong, 2009; Olmos-Solis et al., 2021; Riggall & Postle, 2012; Serences et al., 2009; Stokes et al., 2009). Furthermore, VWM content affects visual processing in a multitude of ways, suggesting at least partially shared functional areas: VWM representations bias concurrent perception of task relevant stimuli (Gayet et al., 2013, 2017; Silvanto & Cattaneo, 2010) and visual stimuli affect saccades differentially depending on their contingency with VWM content (Hollingworth et al., 2013; Hollingworth & Luck, 2009; Olivers et al., 2006; Schneegans et al., 2014; Silvis & Van der Stigchel, 2014; Soto et al., 2005).

Nevertheless, it has been argued that these results do not imply an essential role of the sensory areas. Importantly, Bettencourt and Xu observed in fMRI decoding studies that WM content can sometimes only be decoded from parietal and frontal, however not from occipital sources, when distractors are presented (Bettencourt & Xu, 2016). This finding was taken as evidence for the claim that while early visual areas are involved in VWM, they likely do not play an essential role since we find significant decreases in decodability that are not accompanied by decreases in recall performance. Here, we will argue that a lack or a temporary disappearance of multivariate VWM evidence from early visual brain regions does not imply that they are not essentially required to store and maintain visual memoranda. We do this by highlighting recent findings that demonstrate how memory representations can be dynamically transformed and transferred between brain regions and storage systems. On top of this, we claim that visual memoranda that are actively attended critically depend on recurrent activity in the early visual system.

VWM representations can be “activity silent”

In recent years, there has been accumulating evidence that the visual system has the remarkable ability to transfer visual working memory representations into different codes that are not maintained via recurrent activity patterns and can therefore not be decoded using traditional multivariate pattern analysis (MVPA) techniques (Kamiński & Rutishauser, 2020; Mongillo et al., 2008; Stokes, 2015). These so called “activity-silent” working memory representations are hypothesized to be implemented through changes in short-term plasticity e.g., via calcium kinetics (Mongillo et al., 2008). Part of the information contained in WM would therefore be encoded in short-lived patterns of synaptic weights similar to the encoding of long-term memories, thus giving rise to a state-based, or synaptic theory of working memory. These representations stay hidden from traditional decoding techniques that rely on sustained activity patterns from WM-relevant brain regions. New methodological paradigms had to be developed to investigate these intangible representations. One such approach is to perturb activity silent networks during maintenance periods using strong visual impulses (Wolff et al., 2015, 2017). In the study by Wolff et al., participants were presented with two peripherally presented Gabor patches of which the orientations had to be maintained in memory. A subsequent retro cue indicated which of the two memory items was relevant (the attended memory item “AMI”) and will later have to be compared to a probe stimulus. Importantly, after the cue was presented a large task-irrelevant high contrast stimulus was briefly flashed on the screen. The authors used a distance-based discrimination method to decode orientation representations from the collected EEG data. Interestingly, the neural representation of the cued item was significantly elevated from chance right after the presentation of the high contrast impulse. This was only the case for the relevant cued item (the AMI) and was not observed for the irrelevant un-cued item (the unattended memory item or UMI). These findings suggest that the representation of relevant WM items was encoded in the connectivity pattern of the visual system during the maintenance period. Hence the strong visual transient likely elicited activity that was shaped or “filtered” by this connectivity pattern and could therefore be read out using traditional decoding methods. In a separate EEG study, it was demonstrated that TMS pulses can have comparable effects on WM representations. Rose et al. implemented a similar paradigm by presenting two memory items, a face and a word, of which one was later cued as being the relevant AMI (Rose et al., 2016). The second initially unattended item (UMI) had to be kept in WM and was probed after the AMI. Single TMS pulses were administered during the maintenance period shortly after the first cue defined which item was the AMI. TMS was applied to areas that were previously determined to be functionally implicated in memory maintenance of faces or words during a fMRI experiment. Retro-cues that render memory items irrelevant until a later point in time usually lead to a loss of multivariate evidence for that item. However, the administration of TMS pulses led to a resurgence of evidence that was specifically pronounced for the unattended memory item. These findings provide compelling evidence that WM representations can be stored via short-lived activity silent changes in network connectivity.

The reactivation of latent memory representations using task-irrelevant impulses demonstrates that we cannot rely purely on classical decoding paradigms to infer the functional involvement of specific brain regions in VWM storage. To be clear: we do not claim that the studies by Rose and Wolff proof that memoranda must have been reactivated from early visual cortices since the spatial resolution of TMS and EEG are quite poor. We do want to make the point however that irrelevant stimulation can lead to an increase in multivariate evidence, an effect that is easiest explained by changes in synaptic efficacy, supporting the existence of activity silent codes in the brain. These codes might be present in early visual areas and can be revealed using visual stimulation and likewise can be stored in more parietal regions where TMS can be used to reveal their signatures (Rose et al., 2016; Wolff et al., 2017). When applied to occipital regions shortly before probe comparison, TMS leads to reduced reaction times (Cattaneo et al., 2009). Since TMS pulses are unspecific with regard to stimulus features, it is hard to explain how these pulses can facilitate test probe comparison when representations rely on an active transfer of representations from higher order cortical areas (as was proposed by Xu). Instead, we opt for the simpler explanation that the pulse-induced neural activity is filtered by the stimulus specific synaptic state of the visual system and therefore selectively contributes to the signals that are needed to compare memoranda with the probe. On top of that, it was shown that mental operations performed on VWM items required these items to be retrieved from their dormant activity silent states (Trübutschek et al., 2019). These findings are highly interesting since they delineate under which conditions VWM items critically depend on recurrent activity, e.g., in this case when they need to be manipulated. Moreover, this retrieval effect was not restricted to consciously perceived items but was also observed for items invisible to the observer, implying that even unconsciously perceived items need to be re-activated in order to be manipulated.

Notably any visual impulse should theoretically be able to lead to the re-emergence of a decodable activity pattern. This includes the task irrelevant masks that have been used in fMRI studies like the ones in Bettencourt and Xu (2016) or Rademaker et al. (2019). The reason why mask induced reactivation signals were not observed in these studies is likely due to the short-lived nature of the reactivated signals in combination with the sluggishness of the fMRI signal, the choice of masks used (flickering distractors in Rademaker et al. 2019) as well as the choice of decoding parameters (no time resolved decoding in Bettencourt and Xu 2016). Re-analysis of these datasets might potentially allow us to find similar patterns in fMRI datasets.

While the studies on reactivating activity-silent working memory do not unequivocally demonstrate that the brain relies on early visual areas for VWM storage, they nevertheless urge us to caution when interpreting the lack of evidence from recurrence-dependent decoding paradigms as a lack of memoranda.

Dynamic transformation and transfer of VWM representations

To hide under the radar of spatially constrained multivariate analyses methods, VWM representations do not have to go into activity-silent modes. There is evidence that these memoranda can be transformed into non-visual patterns and transferred to alternative memory systems or states, should the need arise. One of these alternative storages is the long-term memory system (LTM). We have shortly grazed on findings that show how items held in VWM can bias attention and perception (Gayet et al., 2013, 2017; Hollingworth et al., 2013; Olivers et al., 2006; Schneegans et al., 2014; Silvanto & Cattaneo, 2010; Soto et al., 2005). Previously it was shown that these biases reduce with the repeated presentation of VWM items (van Moorselaar et al., 2016). These gradual changes are accompanied by neural markers of WM load like a decrease in the contralateral delay activity and a simultaneous increase in P170 amplitude that is assumed to reflect LTM usage, indicating that VWM memoranda are in fact transferred to LTM (Carlisle et al., 2011; Gunseli et al., 2014; Reinhart & Woodman, 2014). Importantly the reduction in attentional bias reported by van Moorselaar et al. (2016) was reversed towards the end of presentation sequences when a new memory item had to be remembered. This effect was also shown to be accompanied by a re-emergence of contralateral delay activity (Reinhart & Woodman, 2014). The authors concluded that as participants anticipate to update the current memorandum, it likely had to be re-transferred from LTM to an active VWM state. In a follow up study, Gayet et al. replicated and extended these findings by showing that reinstated VWM representations affect perception pre-consciously (Gayet et al., 2019), an observation that links these representations closely to feedforward visual processing (Ress & Heeger, 2003; Supèr et al., 2001; Weiskrantz, 1997). A related line of research further demonstrated how long-term memory representations can spontaneously be re-called into VWM when neuroeconomic influences demand better cognitive control (Reinhart & Woodman, 2014).

Last, we would like to highlight two recent fMRI decoding experiments demonstrating that working memory representations in early visual cortex can be remapped while qualitatively maintaining their underlying multidimensional neural codes. Van Loon et al. (2018) trained a classifier on delay activity while subjects were maintaining a category of AMI’s (e.g., cows or butterflies) (van Loon et al., 2018). Not only was this decoder successfully tested on trials where that category was the AMI but surprisingly showed significant below chance performance when that category was the UMI. This finding hinted at the fact that the UMI was remapped to the opposite region of the neural representational manifold relative to the AMI. Consistent with these findings, Yu et al. (2020) found that orientation representations of UMI’s were systematically rotated relative to their AMI counterparts, demonstrating relevance-dependent remapping of low level stimulus representations in VWM (Yu et al., 2020). Not only are these findings highly informative about the flexible way with which the brain encodes WM representations, but also, similar to the previous section, they remind us to be cautious about the assumptions that we make on how stimuli will be represented neurally, and what we conclude if these assumptions are not met.

The accumulated data not only highlight the flexibility of the interaction between VWM and LTM, but also add to the evidence that active visual memoranda rely on recurrent activity in early visual regions and demonstrate novel ways in which VWM representations are efficiently maintained.

Biological advantages of the sensory storage account

One of the most convincing arguments for the sensory recruitment hypothesis is its biological plausibility. Sharing architectural capacities between visual processing and WM storage reduces redundancy in the brain and reduces the need for additional outsourced subsystems (Postle, 2006; Rademaker et al., 2019). It has previously been argued that this shared resource should lead to detriments in either the perceptual or VWM system (Xu, 2017). Indeed, the contents of VWM have been demonstrated to interact with concurrent perception, however this interaction was frequently shown to be beneficial and has been hypothesized to give rise to attentional effects in perception (Gayet et al., 2017; Hollingworth et al., 2013; Hollingworth & Luck, 2009; Olivers et al., 2006; Schneegans et al., 2014; Silvanto & Cattaneo, 2010; Soto et al., 2005). Visual search paradigms are a prime example of the beneficial effects of WM content of perceptual performance. On top of that, by encoding representations in more robust connectivity-based forms they also become more resilient to interference by distractors or ongoing visual perception. While there are still may unknowns about the interaction between WM and ongoing perception, we dare to hypothesize that this dynamic might be an essential tool for the visual system.

Another important factor to consider when evaluating implementations of neural systems is energy consumption. It is well known that neural activity is energetically costly. Some estimate that this limits the brain to activate a mere 1% of its total neurons at any given point in time (Lennie, 2003). This puts severe limitations on models that pose that WM maintenance is implemented through recurrent activity patterns. Activity silent models however bypass this constraint by utilizing short-term synaptic changes. Since these changes in connectivity do not have to be continuously upheld like recurrent patterns, they are far more energy efficient (Miller et al., 2018; Mongillo et al., 2008). The ability to expend energy only on the maintenance of the currently attended item thus likely provides an evolutionary advantage.

Conclusion

We have briefly reviewed recent findings that highlight the brain’s remarkable ability to transfer VWM representations to alternative memory storages and states. Activity silent representations use short-term synaptic changes to modulate network connectivity that can encode memory items (Mongillo et al., 2008). This allows the memory system to efficiently encode information that cannot be readily read out via traditional MVPA methods. On top of that, EEG and fMRI studies provide evidence that VWM seems to be able to dynamically interact with LTM storage systems, posing similar challenges to traditional decoding studies (e.g., van Moorselaar et al., 2016). The accumulated evidence therefore urges us to be cautious in interpreting the lack of multivariate evidence for VWM content in specific brain regions. As we demonstrated, this lack does not entail that the brain region does not play an essential role in representing relevant information. We propose that in the face of distractors, concurrent visual processing or other perturbations the VWM system recruits alternative memory states/systems to protect memoranda from deterioration. This transformation most likely results in a drop of multivariate evidence for VWM items in early visual areas.

If representations can be transferred to activity silent states or higher order brain areas, which role, if any, does early visual cortex play in VWM storage? Xu acknowledges the fact that one consistently finds VWM representations in early visual cortex (EVC) and that these somehow relate to behavioural performance. In their template matching account, it is stated that feedback signals send to EVC might facilitate the matching of VWM representations to an incoming probe (Bettencourt & Xu, 2016; Christophel et al., 2018; Rademaker et al., 2019; Serences, 2016; Xu, 2017). We would like to propose that this hypothesis extends to all situations in which VWM content undergoes mental manipulations or otherwise needs to become relevant or activated. Furthermore, we would like to claim that these feedback signals not only facilitate manipulation but in fact transfer or reactivate WM representations in EVC making them “visual” working memory representations in nature. In short – the manipulation or maintenance of activated or attended visual working memory representations critically relies on recurrent activity in early visual areas. In the case of mental rotations, this is demonstrated by Trübutschek et al. (2019), van Loon et al. (2018) and Yu et al. (2020) whose findings we have highlighted in Sections 2 and 3. In a WM-based line tracing task, importantly a task without template matching component, it was shown that a mask temporarily erases recurrent WM signals in V1. This activity however reemerges soon after, when behavioural responses are imminent (van Kerkoerle et al., 2017). At this point, it is difficult to provide further evidence for the claim that VWM reactivation generalizes to other experimental contexts since the vast majority of VWM studies, especially the ones investigating the role of visual cortex, use relatively simple delayed match to sample tasks. One possible way to further test this hypothesis would be to instruct participants with modulating VWM representations e.g., via mental rotations (similar to the work by Trübutschek et al., 2019; van Loon et al., 2018; Yu et al., 2020) and test whether such modulations change VWM representations in early visual areas or in higher order brain areas.

Here we reviewed the evidence that the visual system can temporarily and dynamically transform and transfer VWM representations in the face of visual distraction or concurrent visual processing. These findings explain why decoding studies sometimes fail to find multivariate evidence for VWM items in early visual areas and highlight the limitations in the conclusions that we can draw. We propose these alternative storages serve to temporarily protect memoranda. Most importantly however when they need to be put to use, e.g., be matched to samples or otherwise manipulated or activated, they are inevitably required to be transferred back to visual regions in active “visual” form, reigniting active decodable neural patterns.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This project was funded by an ERC Consolidator grant ERC-CoG-863732 to S.V.d.S.