The supramodality “spillover” from neuroscience to cognitive sciences: a commentary on Calzavarini (2023)

ABSTRACT

This is a commentary on Calzavarini (2023), Rethinking Modality-Specificity in the Cognitive Neuroscience of Concrete Word Meaning: A Position Paper 10.1080/23273798.2023.2173789.

This article refers to:

Rethinking modality-specificity in the cognitive neuroscience of concrete word meaning: a position paper

Supramodality as a modality-independent, content-specific processing

The identification of a category-specific object representation within the ventral visual cortical pathway, which we called Object Form Topology (Haxby et al., 2001), challenged the way we used to conceive the cortical underpinnings of cognitive functions. Such an elegant and fine cortical organisation immediately prompted us to believe that Object Form Topology could not be merely restricted to the visual domain but had a more abstract nature. Indeed, subsequent studies in sighted and congenitally blind individuals demonstrated substantially similar patterns of category-specific cortical responses that were independent from both sensory modality and (lack of) visual experience, a property that we named supramodality (Pietrini et al., 2004). Two decades later, the Calzavarini’s position paper (Calzavarini, 2023), provides a comprehensive model that extends the implications and the meaning of supramodality far beyond the mere fields of neuroscience and psychology.

Since our initial study, several studies developed by our own lab as well as by independent centres across the world, have exploited the supramodality concept and provided more and more pieces of evidence to indicate that the (human) brain is, at least to a great extent, organised in a supramodal manner (Amedi et al., 2017; Cattaneo et al., 2008; Heimler & Amedi, 2020; Ricciardi, Bonino, et al., 2014; Ricciardi, Handjaras, et al., 2014). As a matter of fact, the demonstration of the supramodal nature within brain cortical areas and networks that subserve a variety of mental functions – including spatial navigation, action and event representation, language, and even affective processing and social cognition (e.g. Bedny et al., 2009; Benetti et al., 2017; Bonino et al., 2015; Collignon et al., 2011; Klinge et al., 2010; Kupers et al., 2010; Mahon et al., 2009, 2010; Ricciardi et al., 2007, 2009; Striem-Amit et al., 2012, 2016), fostered the hypothesis of a brain organisation “driven by specific computations rather than by specific sensory inputs” began to rise (Heimler & Amedi, 2020).

Historically, brain studies in individuals with congenital lack of either sight or hearing have focused on the plastic reorganisation that indeed follows the (congenital) absence of a given sensory input, a phenomenon known as cross-modal plasticity. While cross-modal plasticity occurs because of the lack of a sensory modality since birth (Bottari & Berto, 2021; Cardin et al., 2020; Castaldi et al., 2020; Frasnelli et al., 2011; Röder et al., 2021), supramodality represents the other face of the medal, as supramodal are those cortical networks that develop and function despite the lack of a sensory modality and therefore they show significant overlapping functional responses across sighted and congenital deprived samples, including blind and deaf individuals. We defined this concept in our first review paper (Ricciardi, Bonino, et al., 2014, p. 65)

if a given feature is also present in sighted individuals, its functional recruitment in congenitally blind individuals has to reflect a more abstract, supramodal representation of a specific content of information, either structurally or semantically, and cannot be simply a consequence of a plastic rearrangement due to the lack of vision.

These studies in congenitally deprived individuals have been indeed essential to comprehend both that sensory (i.e. visual) experience is not a mandatory prerequisite for the brain to develop its morphological and functional organisation, at least to a large extent, and that information content is represented at a cortical level in a “more abstract” format, that is independent from the sensory input: “immune” from sensory experience and “detached” from sensory input, one could say.

The advent of multivariate approaches, greatly supported by AI-based methods, and of models of representational topographies in the brain has favoured a broader and thorough characterisation of the shared neural content between typically developed and congenitally sensory-deprived individuals across different computations or semantic domains (e.g. Mahon et al., 2009; Mattioni et al., 2022; Pietrini et al., 2004; Ricciardi et al., 2013). These novel advances described and decoded the supramodal content at a cortical level – i.e. the information that is represented in a spatially distributed pattern of response – and assessed that the homologies in the functional representation are not limited only to a mere overlap in the topographical localisation of neural responses but they do rely on a common content of the neural responses across sensory modalities and across experimental groups. These shared functional computations do not only enable congenitally deprived individuals to acquire knowledge about different perceptual, cognitive and affective aspects of an external world that they have never seen or heard, but predominantly highlight a more general modality-independent conceptual representation within the human brain.

A cortical proto-organisation is encoded in our genes

The observation of a supramodal functional cortical organisation implies that a prior sensory experience is not mandatory for our brain to develop its magnificent morphological and functional architecture. Consequently, a script for such a development must be encoded in our genes and be present since birth (or even during prenatal life), though it still remains an unresolved matter to what extent the finer cortical functional development of a mature brain is (in)dependent from (any) sensory experience, including that carried by other modalities. Ultimately, after birth different cortical modules and maps develop and refine lifelong and they do so even when a specific sensory experience is absent (Bridge & Watkins, 2019; Castaldi et al., 2020; Cecchetti et al., 2016; Heimler & Amedi, 2020). At the present time, the above issue, that poses fundamental philosophical questions, appears to be impossible to address; indeed, a hypothetical absence of any sensory input at all, should such a condition even exist, by its very nature would not allow to challenge the brain in any manner. The role of experience in the individual representation of the external world has fascinated humanity since the very early days, suffice it to recall the Plato’s Allegory of the Cavern or the Molyneux’s Question that sparkled the still unresolved debate between Rationalists and Empiricists.

To dig deeper within this debate between innate functional organisation and experience-dependent effects, we recently questioned whether brain regions that are responsible for the integrated audiovisual processing – namely, the superior temporal cortex – retain the ability to represent sensory correspondences across modalities even in the case of the congenital lack of either sight or hearing (Setti et al., 2023). To this aim, we took advantage of a naturalistic stimulation paradigm to convey the same stimulus content across different modalities and across samples of typically developed, congenitally blind and congenitally deaf individuals. Our results indicated that the functional architecture of the superior temporal cortex emerges despite the lack of a combined audiovisual input since birth and irrespectively of the differential postnatal sensory experiences. Thus, this cortical area is innately provided with a functional scaffolding to process low-level perceptual features that define sensory correspondences across modalities. At the same time, the refinement of more complex levels of audiovisual skills appears to require a full multisensory experience throughout development. Of note, higher-order characteristics of a given stimulus, such as categorical semantic information, appeared to necessitate experience.

Two main considerations raise from these findings. The existence of a supramodal organisation even in multisensory areas expands previous evidence that large portions of the human neocortex do possess a predetermined morphological and functional architecture – also referred to as proto-organisation –, that forms the scaffolding for subsequent, experience-dependent functional specialisations (Arcaro & Livingstone, 2021; Srihasam et al., 2014). Furthermore, the innate presence of a topographic organisation at a cortical level provides the foundations for the progressive development and refinement of perceptual and cognitive processing. Indeed, previous experimental evidence acknowledges that distinct sensory experiences may cooperate to favour a maturation and refinement of cortical areas in typical development, while detectable adaptations may occur in the functional reorganisation of “sensory deprived brains”, as described above.

We believe this novel perspective is now fully reconciling the idea of an intrinsic, modality-invariant functional organisation that mostly preserves its large-scale organisation, but also adapts and reorganise in response to sensory deprivation, early or later in life. Consequently, researchers could now explore experience-dependent and -independent features of brain development within this hypothesis that large portions of the neocortex possess a predetermined morphological and functional architecture that is of course subjected to the effects of subsequent experience-dependent refinement, a refinement that relies on the integrated cooperation of distinct sensory inputs.

How is unimodal information integrated into more abstract representations

If information is computed in a supramodal manner, one of the critical aspects yet to be solved regards how a given input gets to be “detached” from a specific sensory modality so to be semantically processed or represented, as raised by Calzavarini (2023).

We believe one important distinction must be made. When referring to shared computations in supramodal areas, researchers may often mean the ability of a cortical region to extract specific low/mid-level features from a sensory stimulus (e.g. spatio-temporal frequencies, motion, etc.). Localised cortical modules along the different sensory processing pathways demonstrate computational sensitivity to these features in a modality-independent manner (Amedi et al., 2017; Ricciardi, Bonino, et al., 2014; Ricciardi, Handjaras, et al., 2014). These regions, indeed, include also primary sensory areas able to “capture” some spatio-temporal information of external stimuli in a modality-independent manner (see, for instance, V1 responses to low-level auditory features, such as in Bednaya et al., 2022; Martinelli et al., 2021), possibly contributing to an early multisensory interplay via the representation of sensory correspondences across modalities (e.g. Ghazanfar & Schroeder, 2006). On this aspect, we recently proposed that the deafferented V1 as well may retain its intrinsic, supramodal functioning in the blind brain (Ricciardi et al., 2020).

A different consideration should be made for mid/high-level processing of stimulus features, such as object categories, or when we make sense of a given sensory input (i.e. semantic knowledge). These high-level representations typically rely on more distributed response patterns and are not limited to functionally specialised cortical clusters, that are involved in processing simpler features of stimuli, as properly described in the position paper (Calzavarini, 2023). Thus, one may wonder how the supramodality takes place in the case of these multifaceted representations.

Our previous observations from behavioural studies and brain activity measures in both sighted and congenitally blind individuals during a property-generation task of concrete nouns showed that the overall category-based organisation of conceptual knowledge does not differ across sensory modalities (hearing or vision) and experimental groups (Handjaras et al., 2016, 2017). Furthermore, a wider semantic cortical network correlated with linguistic production and was independent from the stimulus presentation modality and the (lack of) a prior sensory experience. Differences in neural responses between modalities and groups were found only when the analyses were limited to the region-specific content. The combination of information content across a distributed network of regions engaged during the processing of semantic information results to generate a unique, supramodal representation that matches behavioural data and retains the most precise definition of concepts. Conversely, neural representations in limited cortical modules showed category preferences and retained a modality-dependent structure (e.g. “visual” features of object form in lateral occipital cortex). In our opinion, these two distinct levels of stimulus processing – i.e. shifting the description from a smaller to a larger scale cortical representation of the semantic network – do not support the hypothesis that conceptual knowledge relies on specific hubs, but explain how semantic information integrates between brain areas and progresses from a modality-based towards a modality-independent conceptual representation (Deniz et al., 2019; Handjaras et al., 2016; Mahon & Caramazza, 2011; Popham et al., 2021). This modality-independent representation can ultimately be accessed through either distinct bottom-up inputs or executive top-down mechanisms (such as working memory, attention, etc.) (Ricciardi & Pietrini, 2011).

A final remark

To conclude, over the last quarter of a century, the quest on the characterisation of the sensory-deprived brain has offered scientists a unique opportunity to understand to what extent a sensory experience – or the lack of it – shapes the development of brain functional organisation. A converging multitude of results from independent laboratories indicate that, not only, the (human) brain develops its marvellous large-scale architecture in individuals who lack sensory inputs since birth as well, but also that brain organisation is primarily driven by specific property-specific and modality-invariant computations. Overall, these findings support a more efficient and economic brain cortical organisation. This supramodal perspective has immediate implications not only for the understanding of how the brain works but also for how sensory-deprived individuals form a mental representation of the external world. In addition, an expected “spillover” of this supramodal mechanism of stimulus processing has impacted the research on rehabilitation and neuroprosthetic tools, such as the development of sensory-substitution devices (Cecchetti et al., 2016; Maidenbaum et al., 2013). Interestingly, Calzavarini’s position paper is now presenting how supramodality could fertilise and trigger perspective changes also from more theoretical perspectives in philosophy and cognitive sciences, such as semantic knowledge. We could not agree more that this supramodal feature of brain organisation should receive the scientific consideration it deserves.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work has been supported by a PRIN grant (2017_55TKFE) by the Italian Ministry of University and Research to P.P.