Multivariate analysis of brain activity patterns as a tool to understand predictive processes in speech perception

ABSTRACT

Speech perception is heavily influenced by our expectations about what will be said. In this review, we discuss the potential of multivariate analysis as a tool to understand the neural mechanisms underlying predictive processes in speech perception. First, we discuss the advantages of multivariate approaches and what they have added to the understanding of speech processing from the acoustic-phonetic form of speech, over syllable identity and syntax, to its semantic content. Second, we suggest that using multivariate techniques to measure informational content across the hierarchically organised speech-sensitive brain areas might enable us to specify the mechanisms by which prior knowledge and sensory speech signals are combined. Specifically, this approach might allow us to decode how different priors, e.g. about a speaker's voice or about the topic of the current conversation, are represented at different processing stages and how incoming speech is as a result differently represented.

KEYWORDS:

• Multivariate pattern analysis
• representational similarity analysis
• speech perception
• predictive processing

Introduction: aim of this review

The language-processing hierarchy can be broadly organised from bottom to top, i.e. from phonetics, morphology, lexicology, and syntax, to semantics. Errors or misinterpretation of a speech signal can happen all along the hierarchy of speech processing: an unclear phoneme can turn a “cat” into a “bat”, an unclear morpheme can turn the past into the present (“talk-ed” - “talk-s”), and word context can make the difference between life and death (“The chicken is ready to eat.” - leaving ambiguity whether the chicken is going to eat or will be eaten). Especially, under noisy and ambiguous listening conditions, prior experience is a crucial factor in determining the intended meaning and also the specific context in which the sentence is embedded conveys valuable prior cues. One prominent example is the “pop-out” effect (Davis et al., 2005): This effect describes the experience that the identical degraded spoken sentence, which is perceived as unintelligible without prior information, does suddenly “pop-out” and becomes clearly comprehensible after lexical information is provided prior to the hearing. Similarly, in a conversation in a noisy environment, we may interpret an utterance either as “It is very important to teach science” or as “It is very important to teach signs”, depending on the context or topic of the conversation, e.g. either epistemology or road safety, respectively. Such prior expectations can be derived from general statistical regularities on multiple levels, such as phonemes, morphemes, words, syntactic structure, or semantic topics and from how these general statistical regularities appear together (Davis & Johnsrude, 2007; Donhauser & Baillet, 2020; Heilbron et al., 2022; Holmes et al., 2018; Kuperberg & Jaeger, 2016; Levy, 2008; Schmitt et al., 2021; Stilp, 2020).

However, the levels of the linguistic hierarchy are not independent but influence each other mutually under ecologically valid conditions and can hardly be disentangled. If we take the single words “cat” and “bat” as examples for studying phoneme perception, the neural response will inherently be related to differences in semantic processing. Conversely, several probabilistic properties may interfere with other low-level spectrotemporal features such that syntax constrains prepositions to comprise fewer syllables than nouns thereby leading to shorter durations (Miller et al., 1958). In addition, neural processing at both higher but also lower processing stages might be changed depending on bottom-up and top-down information, respectively. These interdependencies across different levels of the processing hierarchy require controlled designs to study processing within and across different levels. Predictive information from the specific conversational situation can make these general priors even more powerful, for example using speaker-specific priors (Krumbiegel et al., 2022).

Cortical brain regions involved in processing speech and language are hierarchically organised (Davis & Johnsrude, 2003; Hickok & Poeppel, 2007). Several fMRI studies have shown a gradient of invariance from low-level acoustic towards abstract high-level non-acoustic processes across brain regions: While responses in primary auditory cortex (PAC) are sensitive to the acoustic properties of the speech input, surrounding regions respond to both speech intelligibility as well as the acoustic distortion, and more distant regions within the superior and middle temporal gyri (STG/MTG), and inferior frontal gyrus (IFG) are sensitive to speech intelligibility, but insensitive to the acoustic form (Davis & Johnsrude, 2003; Okada et al., 2010). Voxelwise modelling based on spectral, articulatory, and semantic features supported hierarchical speech processing further, by revealing spectral features in the core of PAC, mixtures of spectral and articulatory in STG, mixtures of articulatory and semantic in superior temporal sulcus (STS), and semantic in STS and higher-level prefrontal and parietal regions (Heer et al., 2017).

While there are many findings in favour of a hierarchical inference process of speech perception, there is contradictory evidence that points towards a parallel and distributed processing stream of speech stimuli (Hamilton et al., 2021). Yet, gradually descending sensitivity towards low-level features together with staggered temporal integration windows along the cortical hierarchy encourage us to believe that at least the prevailing organisation seems to be hierarchical (Norman-Haignere et al., 2022). With the hierarchical organisation of speech-sensitive regions, the brain is ideally equipped to pass down expectations to improve processing in lower-level brain regions (but see Huettig & Mani, 2016).

From univariate analyses of neuroimaging data using different methods, we know that unexpected, surprising speech leads to an increased brain response, whereas expected speech leads to a reduced response (Blank et al., 2018; Blank & Davis, 2016; Sohoglu et al., 2012). While these univariate response reductions clearly indicate that speech processing depends on prior information, they do not determine the underlying neural mechanisms (Alink & Blank, 2021; Feuerriegel et al., 2021; Todorovic & Lange, 2012). Reduced univariate activity for expected speech could be due to either more efficient and less effortful processing of predicted sounds or to suppressed processing of these sounds (i.e. explaining away; Blank & Davis, 2016; Kok et al., 2012; Murray et al., 2004). Both of these proposals can explain reductions in the magnitude of neural responses for speech sounds that are heard and interpreted as being in line with prior expectations and other similar findings from repetition suppression designs (Aitchison & Lengyel, 2017; Feuerriegel et al., 2021).

In the first part of this review, we discuss the contribution of multivariate methods to deepen our understanding of speech processing, with a focus on non-invasive functional magnetic resonance imaging (fMRI) as a tool with high spatial resolution. In the second part, we discuss the potential of multivariate analysis of brain activity patterns to unravel the neural mechanisms underlying predictive processes in speech perception.

What do multivariate analysis techniques of fMRI data measure?

Traditional univariate analyses of fMRI data evaluate the differences in magnitudes of activation in individual voxels between conditions of interest (Friston et al., 1995), while multivariate approaches investigate how distributed spatial patterns of activity encode information (e.g. multivariate pattern analysis, MVPA; Haxby et al., 2014). In a typical multivariate fMRI study, the goal is to investigate whether it is possible to decode specific stimulus or task conditions from a collection of voxels in a given brain area. For example, one could ask the question if voxels within a brain region represent a particular dimension of the speech hierarchy (e.g. phonemes) and whether this pattern is driven by the acoustic features only or whether additional higher-level and context-dependent processes may shape the neural activity. The term representation is highly discussed across disciplines such as neuroscience, philosophy, and machine learning research (Baker et al., 2022; Jonas & Kording, 2017). Here, we use the term representation to refer to any neural pattern that systematically varies with an external (and possibly internal) stimulus through the features it contains and how these features relate to each other. This correlational approach is often used as an attempt to measure mechanistic principles by linking neural representations with representations derived from theoretical frameworks and computational modelling (Haxby et al., 2014; Yang et al., 2012). With this approach, we use representation in line with the studies we are discussing in this review (Kriegeskorte, 2008). Specifically, we provide an outlook on how new methodological advances may allow us to go beyond the simple correlational approach to investigate representations as underpinnings of mechanistic principles of speech processing.

One advantage of multivariate approaches in comparison to univariate approaches is their increased sensitivity arising from combined information across multiple voxels (Hebart & Baker, 2018; Pakravan et al., 2022). Instead of assuming voxel-independence and considering only a single voxel at a time, multivariate analyses can take the covariance structure of the voxels into account, i.e. the correlation across voxels, so that jointly encoded information is used to gain insights that would otherwise be concealed (Habeck, 2010). Moreover, it is inherent to many types of multivariate analyses to move beyond classical best-parameter estimation and use cross-validation procedures to make predictions and draw inferences about the generalisation to population data (Hebart & Baker, 2018; Yang et al., 2012). For example, semantic generalisation across languages was supported by a simple classification algorithm which was trained on one language and tested on a different one (Correia et al., 2014).

Many different classes of multivariate analysis can be applied to neuronal data, of which the most prominent ones in language research are representational similarity analysis (RSA; Kriegeskorte, 2008) and classification approaches, such as linear discriminant analysis or support vector machines (SVM; Aglieri et al., 2021; Allen et al., 2012; Fisher et al., 2018; Kilian-Hutten et al., 2011). These analyses can be done in whole-brain searchlight analysis, extracting the pattern for every single voxel and its surrounding voxels (Etzel et al., 2013) or with a focus on a given region of interest (ROI; Poldrack, 2007). Multivariate approaches may allow us to shift the focus from the question about which location in the brain is involved in processing speech (where) to the question about which information is represented (what and possibly how).

What do multivariate analysis techniques of fMRI data not measure?

The previous section illustrates how MVPA can serve as a powerful approach to investigating the neural processes underlying the perception of spoken language. However, only with an elaborate research design, it may be possible to make inferences about the quality of the information (i.e. the information content) present at the neural level (Ritchie et al., 2019). Before summarising the contributions of MVPA to our understanding of speech processing, we discuss potential shortcomings of these multivariate approaches. Classification algorithms are information greedy and can pick up on any statistical pattern that is not necessarily related to the intended research question. While decodability indicates that some latent information related to the feature manipulated is processed in a certain brain region, it does not imply that the feature itself is represented there (see Daube et al., 2019 for an example), nor that the information is actually decoded or used by subsequent brain regions and finally relevant for behaviour (Baker et al., 2022; Ritchie et al., 2019). For example, the identity of phonemes could be decoded even though the corresponding percept may have been different (Correia et al., 2015; Lee et al., 2012). So how could we test whether information, as measured with multivariate analyses, is actually relevant for speech perception? Even if patterns of neural activation and the behavioural response patterns correlate (Kilian-Hutten et al., 2011; Preisig et al., 2022), this does not necessarily provide evidence about the quality of information contained in the neural representation and whether the neural representation does indeed affect and initiate downstream behaviour (Baker et al., 2022; Ritchie et al., 2019).

Advantages of similarity-based multivariate analysis techniques

One solution is provided by multivariate approaches based on proximity or distance relationships between conditions, such as RSA. Similarity-based approaches may enable us to make the content of the internal state space more accessible for several reasons (Diedrichsen & Kriegeskorte, 2017; Kriegeskorte, 2008; Kriegeskorte & Kievit, 2013). First, RSA does not only allow us to separate distinct categories based on their (dis)similarity but to assess qualitative features of these categories in terms of their relatedness. Second, RSA can relate continuously parameterised stimuli beyond categorical labels, which is often not the case for classification algorithms in which a class label needs to be discreetly assigned to one category (Kriegeskorte, 2008). A third advantage of RSA is the canonical way of testing representations across various data formats, such as electrode recordings and fMRI measures. Hence, the input for RSA can vary across different spatial and temporal dimensions and even across species. In this approach, possible confounds by the structure of the data itself can be introduced (for an example see Dujmović et al., 2022), but can possibly be avoided by following best practices like cross-validation procedures. Fourth, the most powerful advantage of RSA is that it allows us to make inferences about the underlying processes in the brain during specific manipulations or conditions. Different theoretical models can be transformed into hypothesis-driven predictions of similarity patterns and fit to the neural and behavioural data. Thereby we can assess whether the mechanism or only the resulting output of two systems is similar (i.e. mimicking behaviour, see Dujmović et al., 2022). In summary, results from multivariate approaches provide evidence of varying strength: from the weakest correspondence, i.e. a mere correlation between input and neural response, over a causal role manifest in behaviour, towards a mechanistic approach (Baker et al., 2022; Jonas & Kording, 2017). Thus, with appropriate research designs and carefully worked-out theoretical and computational hypotheses, multivariate methods have the potential to tell us much more than a simple univariate comparison.

In the following, we will review the state of the art in using multivariate approaches in speech perception and next discuss how we could take advantage of these multivariate approaches to go beyond functional localisation and understand the neural mechanism underlying the integration of priors and incoming speech input. With neural mechanism, we refer to how the brain algorithmically performs the computation related to this integration.

What is possible with decoding multivariate fMRI responses in speech and language?

In the following, we will review previous work showing how multivariate approaches can be used to quantify the amount of information that is carried by the spatial patterns of fMRI activity during speech and language processing ranging from the acoustic and phonetic form of speech, to the semantic content of language (Figure 1).

Figure 1. Distributed speech and language processing.

Note. Distributed speech and language processing identified with multivariate pattern analyses along the language-processing hierarchy. The colour coding of brain regions corresponds to the different levels of the speech and language hierarchy (see right). The numbers in the regions correspond to the following studies: 1: Hsieh et al., 2012; 2: Joanisse & DeSouza, 2014; 3: Okada et al., 2010; 4: Evans & Davis, 2015; 5: Pasley et al., 2012; 6: McGettigan et al., 2012; 7: Chang et al., 2010; 8: Obleser et al., 2010; 9: Arsenault & Buchsbaum, 2015; 10: Correia et al., 2015; 11: Archila-Meléndez et al., 2018; 12: Du et al., 2014; 13: Rampinini et al., 2017; 14: Formisano et al., 2008; 15: Bonte et al., 2014; 16: Aglieri et al., 2021; 17: Zhang et al., 2016; 18: Ogg et al., 2019; 19: Kilian-Hutten et al., 2011; 20: Levy & Wilson, 2020; 21: Lee et al., 2012; 22: Preisig et al., 2022; 23: Pereira et al., 2001; 24: Allen et al., 2012; 25: Fedorenko et al., 2012; 26: Hoffman & Tamm, 2020; 27: Carota et al., 2021; 28: Correia et al., 2014; 29: Mahon & Caramazza, 2010.

Acoustic features

On the one hand, we are highly effective in understanding speech despite degraded acoustic properties (e.g. Davis et al., 2005; Evans & Davis, 2015; Remez et al., 1981). On the other hand, subtle differences in the spectral properties of a sound, i.e. its acoustic features, can be crucial for successful speech perception. For example, small frequency shifts contribute to the categorisation of phonemes and differences as small as 75 Hz are perceptible (Flanagan, 2005). The differentiation between up and down sweeps and fast and slow frequency modulations is important for phoneme discrimination, as for example the direction of the frequency modulation at the voice onset distinguishes a /pa/ (rising) from a /ta/ (falling), and the drift rate until voice onset distinguishes /ba/ (fast) from /wa/ (slow). Acoustic features of speech can be successfully decoded with multivariate analyses techniques of fMRI data. For example, fMRI activation patterns from the superior temporal gyrus (STG) and Heschl's gyrus (HG) could be used to classify slow vs fast as well as up and down frequency drifts in complex sound stimuli resembling those of natural sounds, such as speech (Hsieh et al., 2012). These differences in acoustic features were not detectable with classical univariate analyses probably due to the spatial interspersedness of the corresponding representations of direction-selective frequency modulations.

This finding was extended by Joanisse and DeSouza (2014) by investigating the neural correlates of rising and falling, fast and slow frequency modulations on a narrower time scale. While the frequency was modulated within 100 ms for fast changes and 400 ms for slow changes by Hsieh et al. (2012) introducing a confound of duration, Joanisse and DeSouza (2014) kept the stimuli at a constant length while the frequency modulation happened either within the first 50 ms or 100 ms. RSA revealed that the auditory cortex excluding HG encoded direction-specific information of the frequency drifts. In contrast, the rate of the frequency modulation was neither significantly detectable across HG, nor across the auditory cortex. The divergence in the results may stem from methodological changes made to the stimuli but also from the regions of interest under investigation. While Hsieh and colleagues (2012) defined HG based on a functional contrast in combination with individual anatomical considerations, Joanisse and DeSouza (2014) based it on an anatomical atlas only.

Different levels of acoustic degradation, i.e. clear, noise-vocoded, spectrally rotated, and spectrally rotated noise-vocoded versions of the same speech could be distinguished based on the multivariate voxel response patterns in the HG (Okada et al., 2010). Moreover, in accordance with hierarchical speech processing, primary auditory regions were more sensitive to acoustic variation, whereas anterior and posterior regions in STS responded to higher-level intelligibility features of speech and can be described as acoustically invariant. In this study, the highest acoustic invariance was found in the posterior parts of the STS, while previous univariate and later multivariate studies identified the anterior part of the temporal lobe to care least about the acoustic features but the intelligibility (Evans et al., 2014; Scott et al., 2000). In addition to the PAC and regions in the STG, supramarginal gyrus (SMG), inferior frontal, and inferior parietal regions were found to be relevant for discerning speech varying in the acoustic clarity which only became apparent through MVPA (Abrams et al., 2013; Evans et al., 2014; Evans & Davis, 2015). By means of RSA, only the SMG as an extra-temporal region was specifically identified to be sensitive to low-level acoustic details, independent of higher-level features such as syllable identity, speaker identity, or intelligibility (Evans & Davis, 2015). The neural patterns of surface acoustic form in PAC and in the inferior parietal lobule may indicate multiple lines of processing likely coding for different types of information. Future studies will help to disentangle the specific functions of these brain regions in processing low-level acoustic features and whether they form the basis of a complementary or redundant coding system.

In summary, multivariate analyses repeatedly showed that acoustic features of the speech signal are predominantly processed in the primary auditory regions and adjacent areas along the superior temporal plane.

Articulatory features

Humans are able to produce speech sounds by altering the vocal tract and the air flowing from the lungs, up the larynx to be released from the lips. Specifically, sound production involves moving the vocal folds and the resulting glottal pulse rate is filtered by the shape and length of the vocal tract so that specific formant frequencies are enhanced or suppressed. Vowels are produced without obstruction to the air flow leading to clear formant frequencies, while for consonant production airflow modification is essential, causing them to lie in a higher frequency spectrum. For vowel production, the position of the tongue determines whether certain frequencies get amplified leading to formant frequencies. Therefore, the formants take an indispensable role in vowel characterisation. When the first (F1) and second formant (F2) and their ratio were regressed out of the neural signal, the classification accuracy with which vowels could be identified with MVPA decreased (Markiewicz & Bohland, 2016). Also for consonants, the F2 onset frequency and transition magnitude (difference between onset F2 and vowel F2) are linguistically relevant and can integratively map the phonetic distinctions between /b/, /d/ and /g/ (Chang et al., 2010). From an articulatory point of view, consonants are produced by a modification of three features: the place of articulation, manner of articulation, and voicing. In general, those features coincide with acoustic properties. For example, place of articulation can determine 1) labial consonants with ascending formant transition towards the vowel formants, 2) alveolar consonants with descending formant transitions, and 3) palatovelar consonants with descending F2 and ascending F3 transition (Nearey, 1998). Many of the corresponding articulatory neural patterns could be mapped with multivariate approaches contributing to a greater understanding of speech processing (Aglieri et al., 2021; Archila-Meléndez et al., 2018; Arsenault & Buchsbaum, 2015, 2016; Bonte et al., 2014; Chang et al., 2010; Correia et al., 2014; Formisano et al., 2008; Gardumi et al., 2016; Kilian-Hutten et al., 2011; Lee et al., 2012; Levy & Wilson, 2020; Ley et al., 2012; Obleser et al., 2010; Preisig et al., 2022; Rampinini et al., 2017; Zhang et al., 2016). In general, vowel representations seem to be more robustly identified with MVPA than consonant representations (Zhang et al., 2016), probably because vowels can stand for themselves while consonants unfold their full articulative spectrum only in combination with a vowel and are influenced by coarticulation and therefore more variable. Moreover, it has been shown that consonant and vowel-specific processing relied on distinct sparsely overlapping areas in the STG encoding phonetic features at different levels of abstraction (Chang et al., 2010; Obleser et al., 2010).

In the following, we will describe how multivariate pattern approaches were used to investigate the perception of consonants and vowels.

Consonants

As explained above, the place of articulation is a crucial feature in consonant discrimination. Chang et al. (2010) found multivariate neural patterns in the pSTG that encoded consonants’ place of articulation characterised by the F2 onset frequency and transition magnitude. A subsequent MVPA study identified brain regions that process specific articulatory features independent of phonemic category and acoustic variation and extended the knowledge through whole-brain coverage which brought about the involvement of frontoparietal regions and the insula. Not only for the place of articulation but for each of the two remaining articulatory features (voicing and manner of articulation) distinct patterns were found. The different articulatory features partly overlapped in anterior temporal lobe and inferior postcentral gyrus, thereby indicating the encoding of common higher-level features (Correia et al., 2015). The contribution of the sensorimotor systems (including the postcentral gyrus) and their necessity for speech perception has been a long-standing debate (de Zubicaray et al., 2013; Hickok, 2014; Kemmerer, 2015; Schomers & Pulvermüller, 2016). Different hypotheses about the integration between speech perception and speech production have been introduced. Proponents of the motor theory of speech perception postulated that the acoustic signal is decoded as a function of articulatory motor patterns (Liberman et al., 1967; Liberman & Mattingly, 1985). Moreover, it has been suggested that covert rehearsal of the articulatory features generates a synthesis of perceptual input with procedural motor commands resulting in enhanced intelligibility (Scott & Johnsrude, 2003). Others have argued that the connections between the perceptual and procedural systems serve the refinement of speech production during the processing of speech input which is not necessarily mutually exclusive (Rampinini et al., 2017). Especially in studies investigating consonant representations, the question around the involvement of sensorimotor areas yielded some controversies. On the one hand, neural patterns from the temporal cortex, but not the motor or premotor cortex, reflected the three distinct articulatory features (Arsenault & Buchsbaum, 2015, 2016). On the other hand, activity from premotor and motor areas could be used to distinguish between manner of articulation (i.e. stop consonants versus fricatives) and without further training the same classifier generalised to differentiate between places of articulation (Archila-Meléndez et al., 2018; Correia et al., 2015). Under noise, consonant categorisation based on multivariate fMRI pattern was shown to be more robust in the left ventral premotor cortex and Broca's area than in bilateral temporal regions (Du et al., 2014). The enhanced neural activity under noisy conditions could only be differentiated from alternative explanations (i.e. attention, task demands) with RSA indicating that it rather reflects shared phoneme-specific information. A recent review that scrutinised the role of the sensorimotor system in speech perception, proposed that lacking motor involvement is likely due to confounds introduced through excessive scanner noise and prompted motor responses (Schomers & Pulvermüller, 2016). This is not entirely in line with a linear relationship between noise and the involvement of premotor regions, but with an inverted U-shape account of noise. An elaborate study design using RSA to reflect different levels of abstraction showed that the left motor cortex is only related to high-level features of speech such as syllable identity and consonant–vowel structure, pointing to a hierarchy-dependent function of sensorimotor integration (Evans & Davis, 2015). It is conceivable that predictive mechanisms form the basis for this overarching functional contribution such that for example, the motor system may help in dynamically predicting the signal according to the expected frequencies. Future studies may attend to the challenge and unravel the mechanisms underlying this integrative, possibly predictive, circuitry.

Vowels and voice

Vowel and voice perception both rely on the same acoustic dimensions, i.e. the frequency spectrum of speech sounds. In a seminal study, Formisano and colleagues decoded both distinct speaker as well as distinct vowel identities from multivariate fMRI patterns (Formisano et al., 2008), a finding which was later repeatedly replicated (Aglieri et al., 2021; Bonte et al., 2014; Gardumi et al., 2016; Zhang et al., 2016). None of these studies found statistically significant contrasts between different speakers and different vowel conditions with conventional univariate analysis. Importantly, even with advanced techniques like ultra-high field fMRI no effect was found (Gardumi et al., 2016; Zhang et al., 2016). In general, classification of speaker and vowel identity relied on distinct but overlapping regions in the superior temporal sulcus and gyrus (STS/STG; Formisano et al., 2008). Also distinct patterns for common articulatory features such as front vowels (/i/ and /e/) in comparison to back vowels (/u/ and /o/) for naturally coarticulated syllables were identified with MVPA, most prominently in aSTG (Obleser et al., 2010). While vowel information was represented bilaterally, speaker information was found to be right lateralised (Formisano et al., 2008). Interestingly, classification of speakers and vowels based on fMRI response patterns was dependent on the task and hence showed that information obtained from fMRI response patterns depends on top-down enhancement (Bonte et al., 2014). Neural classification accuracy which distinguished voices based on the BOLD signal of anterior STG and IFG, correlated with behavioural voice recognition performance across participants, pointing to the importance of extra-temporal higher-level regions in voice recognition (Aglieri et al., 2021). For the perception, production, and imagination of vowels, similar brain regions, e.g. IFG and MTG, yet distinct non-overlapping contiguous clusters within these regions were identified to be engaged with multivariate phoneme classification (Rampinini et al., 2017). This speaks for distributed patterns representing constant multimodal integration through reciprocal connections facilitating the dynamical neural processing of perception, production and imagery. Yet the exact mechanisms of these connections remain to be studied.

Moreover, it is still an open question whether the auditory processing stream is specifically tuned to human voices like it has been suggested for conspecific vocalisations of animals (Perrodin et al., 2011), or whether all acoustic stimuli have the same relative importance. For the decoding of different voices in comparison to instruments, which both vary in their timbre, voxels along the posterior to anterior STG and parts of middle temporal gyrus cloud be exploited by a linear discriminant classifier (Ogg et al., 2019). Further multivariate classification analysis showed more privileged processing of human voice sounds, meaning that the neural response to conspecific vocalisations was well-defined and most reliably distinguished from every other living and non-living sound category, in the right middle temporal gyrus (Lee et al., 2015). Yet, from this analysis, it remains indistinguishable which features of the voice were relevant for the differentiation and whether it relies rather on low-level acoustic features or on higher-order identity abstraction. Voxels relevant for voice-identity discrimination were only found in a small cluster close to the left PAC indicating that voice-identity classification relies on distinctions in low-level acoustic features (Ogg et al., 2019). Despite their intermingled features vowel identity and voice identity could be reliably decoded indicating an abstract representation of those categories. It stands to question how speaker identity and vowel identity are mapped from the variable acoustic signal to abstract category-level representation. We will discuss the neural pattern of this abstraction to discrete behaviourally relevant representations in the next paragraph.

Categorical features

Multivariate approaches were also repeatedly used to test the neural correlates of categorical speech perception, i.e. the observation that listeners perceive speech sounds that vary along a continuum as categorical so that sounds within a category sound more similar than equidistant sounds from different categories (Kilian-Hutten et al., 2011; Lee et al., 2012; Levy & Wilson, 2020; Preisig et al., 2022; Zhang et al., 2016). Those abstract categorical representations of phonemes were revealed in the STG. Firstly, based on ECoG recordings located in the lateral part of the pSTG, the three phoneme categories /ba/, /da/, and /ga/ were successfully classified and the resulting category boundaries from the neurometric identification function reproduced the corresponding psychometric identification function. When the results were extended to fMRI, consonants could be reliably decoded from voxels across the STG (Obleser et al., 2010). In the pSTG however, phoneme classification was not preferentially decodable in comparison to the noise versus speech sound categorisation. Potentially, in contrast to ECoG recordings with high spatial resolution, the coarse BOLD signal in fMRI can detect higher-level, more stable representations of phoneme categories in the higher-level aSTS region. Outside of the temporal lobe, consonant–vowel syllables along a /ba/–/da/ continuum evoked distinct neural activity patterns between the two perceptual categories (/ba/ vs /da/) in Broca's area and the left pre-supplementary motor area (Lee et al., 2012). Interestingly, multivariate and univariate analyses revealed different regions for categorical perception when applied to the same data set, though using different contrasts: a multivariate classifier that was trained on the perceptual category boundary between /ba/ and /da/ revealed Broca's area, whereas a univariate adaptation analysis contrasting between-category versus within-category pairs revealed involvement of the SMG (Lee et al., 2012; Raizada & Poldrack, 2007). Hence the two different analysis approaches seem to identify different functions of Broca's and SMG in speech perception, with phoneme identification versus change detection or discrimination, respectively (Lee et al., 2012).

In accordance with the representation of phonetic features at different levels of abstraction, within-category distinction between either two vowels or two consonants based on voxels in the STG generalised to another within-category distinction (i.e. from /ba/ vs /da/ to /ba/ vs /ma/; Zhang et al., 2016). The discussion at which level of the speech processing hierarchy the transformation of the veridical sound to the abstract category level representation takes place is still debated. Already at the level of the PAC, an alteration of the veridical speech signal represented information beyond simple acoustic features: multivariate classification of voxels in HG revealed that between-category classification was more accurate than within-category classification even though the difference between the vowel stimuli was perceptually equispaced (Levy & Wilson, 2020). Thus, the response patterns did not follow the linear properties of the articulatory features but rather reflected category boundaries. Also, after learning a linguistically-irrelevant, artificial category boundary, which relied on pitch information, perceptual representations of these novel sound categories could be decoded from HG, planum temporale, and regions along the STS - whereas this categorical information was not present when the identical stimuli were classified before learning (Ley et al., 2012). Similarly, Kilian-Hutten et al. (2011) found that SVM could exploit the neuronal activation in Heschl's gyrus and sulcus, and the anterior planum temporale to classify the perceptual category of the identical ambiguous phoneme sound (/aba/ versus /ada/). As the acoustic features of the stimuli were physically identical, the spectrotemporal characteristics cannot serve as an explanation for the differences in perception, nor in the BOLD signal. The authors interpreted this finding as an argument against a hierarchical processing organisation of speech sound, because the categorical perception, which supposedly is a more abstract, higher-level feature involving interpretation, was decodable from low-level auditory regions. An alternative interpretation within the framework of predictive processing might be that representations measured in the auditory cortices are influenced by the higher-level predictions, which are conveyed from higher to lower-level auditory regions (for similar reasoning in the domain of face processing see Schwiedrzik & Freiwald, 2017). Both of the above studies restricted their analysis to the temporal lobe region or even to the HG. Hence, the investigation of activation in and connectivity to higher-level cortical areas which convey categorical priors to earlier regions would help to resolve these two interpretations.

Evidence for influences on categorical phoneme perception from higher-order cortical regions was provided by a recent study that used MVPA to compare the pattern of unambiguous /da/ and /ga/ stimuli to the pattern of ambiguous stimuli at the individual psychometric boundary (Preisig et al., 2022). The ambiguous stimulus was manipulated to be biassed towards one of the two categories by exploiting the mechanism of binaural integration, i.e. presenting the acoustically carved out F3 of either the /da/ (high F3) or /ga/ (low F3) to the left ear and the ambiguous 50% morph of /da/ and /ga to the right ear leading to a biased percept. Critically, there were trials in which the ambiguous stimulus was acoustically warped towards one category, but the opposite category was perceived, which allowed disentangling the intermingled influences of low-level acoustic features (low vs high F3 stimulus) and incongruous top-down higher-level categorical perception (/da/ vs /ga/) of these consonant–vowel speech sounds. While the perceived category in ambiguous conditions could be decoded from distributed clusters, the veridically presented low-level acoustic features could not be distinguished based on their neural response pattern. Remarkably, many higher-order cortical regions seemed to be involved in the categorical perception of consonants. In concordance with previous studies, the classifier trained on the unambiguous stimuli generalised better to the corresponding perceived category than to the acoustically presented stimuli in the auditory parabelt, pars triangularis, and extended motor areas. The authors conclude that it is likely that higher-order cognitive processes impact the neural responses in auditory cortices probably in a predictive manner (Preisig et al., 2022). This interpretation is in line with work in the domain of visual perception, showing that multivariate approaches could accurately classify the shape and orientation of an object (i.e. higher-level features) based on the lower-level primary visual cortex (V1), which is considered to be shape- and orientation-invariant (Jehee et al., 2011; Williams et al., 2008). These low-level activity findings were also explained as reflecting modulations of feedback from higher-level regions. Hence, similar mechanisms of how sensory information is represented based on top-down information such as categories might take place in auditory regions. It remains to be determined whether this effect is indeed driven by a top-down enhancement or an effective categorical representation at the level of PAC.

Many studies have provided evidence of a categorical perception that maps acoustic features nonlinearly to abstract categories. The question at which level of the speech hierarchy this transformation emerges remains elusive. Controlling for alternative factors driving between-pattern variance like low-level sound structures such as loudness, spectral centroid, pitch, or harmonic-to-noise ratio are necessary to disentangle general low-level acoustic processes from category-specific processing (Giordano et al., 2013).

Syntactic

One integral part of language is the use of syntactic principles to form more complex expressions consisting of a systematic combination of words or morphemes. While it has been shown that the brain is perfectly equipped to facilitate predictive integration of the linguistic constituent structure through tracking information at different levels of the grammar-based linguistic hierarchy (Ding et al., 2016), multivariate analysis provided additional insights into the neural patterns of semantic and syntactic uncertainty that were decodable from activity in IFG, STG and MTG (Pereira et al., 2001). Voxels in these areas predicted the response to ambiguous sentences in which ambiguity was resolved in contrast to sentences remaining unambiguous. Further evidence for the syntactic processing in frontal and anterior temporal regions was provided by an MVPA study that disentangled the grammatical pattern of dative and ditransitive sentences, which only differed syntactically in the prepositional phrase while the number of arguments, surface complexity, frequency of occurrence, and the event of the construct remained constant (Allen et al., 2012).

Other findings place importance on more posterior regions of the temporal cortex and the IFG for syntactic information processing only (Ding et al., 2016; Fedorenko et al., 2012). Again solely multivariate, but not univariate, analysis was able to uncover that these regions are involved in syntactic processing. However, no regions in the language network seem to process exclusively syntax-specific information but seem to be concomitantly involved in the representation of lexical information (Fedorenko et al., 2012). Also, advanced analysis techniques, such as the identification of neural patterns with neural network language models found syntactic representations only in temporal language regions that were also implicated in other processes (Heilbron et al., 2022). The interleaved representations may reflect the interdependence of the different aspects of speech and language processing that rely on the constant integration of dynamic top-down and bottom-up information.

Semantics

The highest level of the speech processing hierarchy that we discuss in this review is the semantic abstraction. To successfully convey a message via auditory communication, the main goal is to relate the acoustic input to meaning. Semantic abstraction can happen already at the level of morphemes, e.g. the prefix “un-'” signifies negation, but is most commonly studied at the level of words or sentences. Homonyms are phonologically similar words that possess two meanings (e.g. bat as the animal or the wooden stick), of which the intended one only becomes apparent through context. They constitute a useful class of words to investigate predictive processes in semantic perception because their acoustic input is identical, hence low-level spectral influences are controlled. Semantic information of single homonyms could be reliably decoded from the IFG, anterior temporal lobe and medial temporal gyrus (Hoffman & Tamm, 2020), whereas univariate analysis did not show increased activity in the temporal lobe in response to homonyms.

Different explanations for the emergence of semantic-linguistic meaning have been proposed (Carota et al., 2021). On the one hand, distributional theories posit that the semantic meaning of a word is derived from the frequency of its co-occurrence with other words, i.e. the conceptual knowledge through context of language usage (e.g. lake and swim). On the other hand, taxonomic theories advocate that meaning is derived from a hierarchical abstraction of the conceptual properties of words grouped into related categories (e.g. cat and whale). By means of RSA, the left IFG was found to represent semantic meaning in the distributional sense, while pMTG captured the semantic constructs as predicted by the taxonomic account (Carota et al., 2021). Additionally, the pattern in the angular gyrus related to semantic concepts irrespective of theory. However, in this study the semantic concepts were only evoked by visual presentation. Multivariate patterns of semantic meaning are mostly investigated with written, not with spoken, language stimuli (Carlson et al., 2014; Carota et al., 2017; Mitchell et al., 2004; Mitchell & Cusack, 2016; Pereira et al., 2018), probably because this approach does not hold the disadvantages of the confounding scanner noise. However, recent studies using MVPA suggest largely congruent semantic representations between auditory and visually presented language. This is underpinned for example by a semantic model forming accurate predictions irrespective of modality (Deniz et al., 2019) and an MEG study showing word-specific neural activity shared between modalities and subjects (Arana et al., 2020). Semantic representations evoked by auditory input, which was based on distributional word co-occurrence statistics, yielded an intricate semantic, cortical map persistent across individuals. This semantic map spanned prefrontal, temporal and parietal regions and showed domain specificity that was largely mirrored between hemispheres (Huth et al., 2016). In bilingual speakers, it has been shown that semantic representations are highly abstract and generalised across different languages because the meaning of a spoken word could be decoded in one language based on a SVM classifier which was trained on the other language (Correia et al., 2014). Univariate contrasts could not capture this abstractness which is mirrored in the vast extent of involved brain regions ranging from the clusters in different regions of the temporal lobe, over clusters in the angular gyrus, to the insula, postcentral gyrus, and occipital cortex. Many of these regions were also reported for semantic processing in visual stimuli, thus further supporting the thesis that, indeed, semantic concepts seem to be robust not only across different languages but also with respect to different sensory modalities.

Conceptual similarity to a preceding word influenced the activity pattern in distributed areas of the brain, primarily in parietal and frontal regions (Mahon & Caramazza, 2010). Yet, the relatedness of two words could also be classified based on V1, even though participants were only presented with auditory spoken words, probably due to evoked visual associations. The previous findings highlight the increasing involvement of different distributed systems at the top level of the language hierarchy resulting in a multimodal comprehensive percept. This could be implemented by a distributed set of fragmented supramodal feature nodes that collectively form the semantic concept of an object (Kivisaari et al., 2019). The modulation of neural activity as a function of semantic similarity to the preceding context suggests that it is likely that predictive mechanisms of concept similarity influence the perception of speech stimuli. Since successful semantic classification was obtained from neural responses in the left lateral temporal cortex, semantic predictions may even be propagated down the hierarchy to inform peri-auditory regions (Mahon & Caramazza, 2010). This is in line with context-dependent semantic processing (Deniz et al., 2021). Altogether, MVPA revealed distributed networks of semantic processing that ranged from temporal to frontoparietal regions. Future studies which focus on auditory stimulus presentation may shed light on the predictive nature of semantic speech processing.

How can we use multivariate analysis to tackle open questions (in predictive hierarchical processes) in speech and language?

Information contained in spatiotemporal patterns of neural activity and analyses of representational content (Kriegeskorte & Kievit, 2013) might help us to specify the neural mechanisms by which listeners combine prior knowledge and incoming speech signals. Specifically, this approach allows to measure representational content in the speech-sensitive brain areas and to decode whether the expected or the unexpected part of a speech stimulus is represented and how these representations may be linked to perception (Blank et al., 2018; Blank & Davis, 2016). The key idea here is to distinguish between mechanisms via which prior expectations and acoustic sensory input might be integrated. The commonly observed reduced activity for expected speech might result from a reduced surprise or error signal, as suggested by predictive coding theories, in which neural activity represents the difference between heard and expected signals (prediction error). Alternatively, neural activity might be reduced because it is less noisy and more directly represents the current perceptual experience, i.e. the posterior in Bayesian terms (Aitchison & Lengyel, 2017). While both prior knowledge and increased sensory clarity can improve perceptual experience in a similar way and hence result in similar enhanced representations of the posterior, prediction error representations should theoretically be less informative for speech that clearly matches prior expectations (Blank et al., 2018; Blank & Davis, 2016).

We applied this approach in two previous studies (Blank et al., 2018; Blank & Davis, 2016), in which we combined informative and uninformative written text with degraded speech while measuring the representational content of STG responses with fMRI. Firstly, we showed that both increased sensory detail and informative priors improved behavioural word report in an additive way, but that corresponding neural representations in the pSTG showed an interaction: There was a greater pattern similarity between clearly articulated, partly-concordant stimulus pairs presented after uninformative text than for those that were degraded. However, if an informative prior was available the pattern similarity between stimulus pairs became weaker for clear than for degraded speech. This shows that there was less shared information between stimulus pairs, hence once prior knowledge is available, expected sensory information can be cancelled out when the stimulus is presented (Blank & Davis, 2016). This finding was replicated (Sohoglu & Davis, 2020) and rules out both a pure bottom-up account (without influence of prior knowledge, see Norris et al., 2000) and sharpening theories of speech perception according to which effects of prior knowledge and sensory quality should be additive (McClelland & Elman, 1986). Similarly, misperception, induced by partially deviating prior information as written words preceding degraded spoken words, was explained by multivariate representations of prediction errors (Blank et al., 2018). In sum, these two RSA fMRI studies converge in showing representations of prediction error in the STS/STG.

Moreover with a gradual implementation of prior expectations by modulating the transition regularities between events it could be shown with MVPA, that aberrant non-veridical auditory percepts such as auditory hallucinations or tinnitus could result from a dysfunction of overweighting internal priors (Alderson-Day et al., 2017; Partyka et al., 2019; Partyka et al., 2019). By manipulating prior expectations at different levels of the speech hierarchy meaningful insights about (non-)aberrant speech processing may emerge. For example, the long-standing debate about pre-activated predictions imposed by phonological constraints of sentence context (e.g. by the indefinite article a / an) on the following word could benefit from MVPA (DeLong et al., 2005; Yan et al., 2017). By means of RSA, the neural response after the constraining context could be tested to be either more or less similar to a word overlapping on different levels with the expected word (e.g. “an ace”, acoustic match: ice, syntactic: tree (noun), semantic: trump card (synonym)). Furthermore, it is expected that the similarity between the “matching words” and the anticipated word decreases when the constraining context is modulated to be more uncertain (i.e. presenting a morph between a / an). Additionally, it would be interesting to see whether the neural response after a mismatch depends on the similarity of the neural pre-activation to competing alternatives.

To assess this full range of incrementally unfolding predictions at every level of the nested speech processing hierarchy, methods with high temporal or spatial resolution may enable us to compare the data against predictions from generative models of speech processing that deploy principles of predictive processing (Su et al., 2022).

Limitations of MVPA in fMRI

Another consideration is that the different spatial scales, which presumably support hierarchical abstraction, range from small circuits within a specific region to broad macro-networks such as the default mode network (Yeo et al., 2011). When considering fMRI analysis, the scale at which neural signals are recorded usually lies more or less around a few cubic millimetres per voxel. MVPA can range from single voxels, local searchlight analysis variable in size, over predefined regions of interest to whole-brain analysis (Jolly & Chang, 2021) and each of the approaches balances the trade-off between detecting more local or distributed network patterns. Analyses using a small neighbourhood of voxels in a searchlight analysis detected processes at a smaller spatial scale whereas whole brain analysis revealed processes that generalised well over different task and populations like found for language-invariant semantic representations (Correia et al., 2014; Jolly & Chang, 2021). Thus, the choice of the spatial scale should depend on the scale of the investigated phenomenon.

Additionally, it has been shown that MVPA is sensitive towards noise correlations across BOLD signals assigning greater importance to vowels with similar noise patterns in classification (Bejjanki et al., 2017). With the coarse spatial resolution of fMRI and the slow nonlinear hemodynamic response function, real “second-order multivariate effects” based on the covariance matrix of voxel activity may not be captured. The usage of geometry-aware distance measures such as the Riemannian distance, may more robustly capture multivariate neuronal patterns (Pakravan et al., 2022; Shahbazi et al., 2021).

Future outlook: using neural networks to decode pattern of language?

Technical advances with ultra-high field fMRI have facilitated neuronal layer-specific analysis which could disentangle predictive bottom-up and top-down processes in the visual and somatosensory domains and has yet to be applied to auditory processing (Kok et al., 2016; Muckli et al., 2015; Yu et al., 2019).

The usage of RSA with a fusion of different data acquisition methods, e.g. MEG combined with fMRI (for an example see Lowe et al., 2020), to investigate the informational content in high spatiotemporal may bring the field forward (Cichy et al., 2016). With an advanced reconstruction estimate based on a combination of EEG with a hierarchical prior of fMRI data, classification accuracies were comparable to those from ECoG (Yoshimura et al., 2016). In the future, a continued combination of multivariate methods of high temporal and spatial resolution may establish fruitful insights into the spectro-temporal hierarchy of predictive speech processing.

In addition, recent work demonstrated how neural networks (NNs) are able to reproduce human auditory behavioural patterns when trained on human sounds and tested on a variety of auditory tasks. This approach can help us to derive testable hypotheses about the mechanisms underlying speech processing and calls for a continued investigation of whether the mechanisms may converge between NNs and human neural representations (Kell et al., 2018; Saddler et al., 2021). Indeed, it has been found that the better the performance of a NN, the better it mapped onto neuronal data (Schrimpf et al., 2021). The representational hierarchy of speech processing was apparent in NNs with intermediate network layers mapping best to neural activity in PAC, while later layers predicted activity in adjacent regions along the temporal plane best (Kell et al., 2018). This representational gradient was demonstrated to be independent of the learning modality, i.e. supervised or unsupervised learning in the NN (Wang et al., 2019, 2022). In line with these findings, the combination of deep neural networks and RSA showed that the informational content of the final network layer represented concepts at the sentence level similar to distributed temporofrontal regions (Acunzo et al., 2022).

In another approach, different language models based on word embeddings in combination with MEG were useful to probe the neuronal hierarchy of speech processing (Broderick & Lalor, 2020; Heilbron et al., 2022). Specifically, MEG recordings revealed a temporal hierarchy of speech features so that acoustic processing was followed by phonemic and syntactic information, which was processed earlier than lexical and semantic predictions (Heilbron et al., 2022). In addition, the propagation of high-level predictions towards lower levels and the mutual existence of prediction and prediction error in a predictive coding manner were observed (Broderick & Lalor, 2020; Heilbron et al., 2022). Moreover, competing next-word predictions from a neural word embedding model could be mapped to neural pre-stimulus activity, which after the onset of the next word was amplified by the amount of the surprise predicted by the network model (Goldstein et al., 2021).

Similar approaches based on fMRI data may be beneficial to disentangle the influence of bottom-up prediction error and top-down predictions, such as the successful discrimination of a better-performing hierarchical long-short-term memory model in comparison to a non-hierarchical language model (Schmitt et al., 2021). As we have discussed above, a correspondence between NNs and neural activity can arise albeit the mechanistic principles of the underlying systems deviate (Baker et al., 2022). How can we take advantage of NNs to go beyond simply observing correspondences with neural activity? To obtain an explanatory mechanism for neural activity from NNs, incrementally varying the training input stimuli in a controlled manner may help to examine the degree of algorithmic equivalence. Best practices for neuroscientific research with NNs such as an expansion of the out-of-sample testing space (to other behavioural tasks) as well as biologically-plausible models can serve as helpful measures (Saxe et al., 2021; Schyns et al., 2022). However, constraining a NN's biological plausibility might not always be obligatory since idealised models can help to understand the mathematical learning principles and infinite-size networks are paradoxically more interpretable (Ma & Peters, 2020; Schyns et al., 2022). Tools of explainable AI (XAI) are a promising new approach for filling the gap of interpretability to uncover meaningful patterns in NNs (Samek et al., 2021; Thomas et al., 2022). Applying these tools (for an example on EEG data see Cui & Weng, 2022) together with considering not only the performance of NNs but also their learning trajectories and error landscapes may be beneficial to compare artificial and human mechanisms.

In sum, computational modelling in combination with multivariate analysis techniques of brain data has the potential to improve our understanding of the mechanisms underlying predictive processes in speech processing.

Acknowledgments

This work was funded by the Emmy Noether program of the Deutsche Forschungsgemeinschaft (German Research Foundation; Grant No DFG BL 1736/1-1 to HB).

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by Deutsche Forschungsgemeinschaft: [grant no DFG BL 1736/1-1].