Better together: integrating multivariate with univariate methods, and MEG with EEG to study language comprehension

ABSTRACT

We used MEG and EEG to examine the effects of Plausibility (anomalous vs. plausible) and Animacy (animate vs. inanimate) on activity to incoming words during language comprehension. We conducted univariate event-related and multivariate spatial similarity analyses on both datasets. The univariate and multivariate results converged in their time course and sensitivity to Plausibility. However, only the spatial similarity analyses detected effects of Animacy. The MEG and EEG findings largely converged between 300–500 ms, but diverged in their univariate and multivariate responses to anomalies between 600–1000 ms. We interpret the full set of results within a predictive coding framework. In addition to the theoretical significance, we discuss the methodological implications of the convergence and divergence between the univariate and multivariate results, as well as between the MEG and EEG results. We argue that a deeper understanding of language processing can be achieved by integrating different analysis approaches and techniques.

KEYWORDS:

• ERP
• representational similarity analysis
• Animacy
• predictive coding
• N400
• P600
• evidence accumulation

Acknowledgments

We thank Trevor Brothers and Victoria Sharpe for helpful discussion. We also thank Jeff Stibel for his support of Drs. Kuperberg and Wang.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

1 In the MEG literature, the N400 is sometimes referred to as the M400 or N400m.

2 As noted under Methods, there were small differences between the animate and inanimate nouns in their length and frequency. Effects of word length can influence earlier ERP components (Hauk & Pulvermuller, 2004; Osterhout et al., 1997; Wydell et al., 2003), and differences in frequency can modulate the N400 (Dambacher et al., 2006; Payne et al., 2015; Van Petten & Kutas, 1990). These differences, however, are unlikely to explain the absence of a univariate effect of Animacy between 300-500ms.

3 This may include (a) spontaneous ongoing neural activity, (b) activity stemming from other physiological sources, and (c) noise from the environment or from our measurement instruments (Luck, 2014b). We also note that while spatial similarity and event-related analyses are sensitive to many of the same sources of noise, there are some differences: In an event-related analysis, spontaneous neural activity that is not phase-locked to an event of interest is always considered “noise”. In a spatial similarity analysis, however, non-phase-locked activity can, under certain circumstances, contribute to increases in spatial similarity. For example, if the phase value of a waveform measured at channel 1 varies across trials, then the event-related response at this channel will always be small. However, if the relative differences in amplitude across the three channels (e.g. 1 < 2 < 3) are consistent across trials, then this will give rise to an increase in spatial similarity.

4 This is because without cross validation, estimates of the true Euclidean distance become increasingly more inaccurate (larger) with increases in noise. As discussed above, this is also true of Pearson’s distance estimates. However, the Pearson’s distance between two entirely random patterns has a maximal value of 1 (or, equivalently, a Pearson’s r value of 0), regardless of noise level. In contrast, the Euclidean distance is non-negative and unbounded. Therefore, without cross-validation, the noise inflation of Euclidean distance is greater than that of the Pearson’s distance. Therefore, without cross-validation, Euclidean distance values are far less interpretable than Pearson’s distance values (see Guggenmos et al., 2018; Walther et al., 2016 for discussion).

5 We also carried out supplementary analyses to check that the event-related and spatial similarity effects were similar between the two EEG datasets. For both types of analyses, the pattern of findings across the two datasets was indeed qualitatively similar.

6 Activity is also produced by tangentially orientated dipoles within the walls of a single sulcus. However, these dipoles frequently face in opposite directions, canceling each other out, and so they are not detected at the head surface by either MEG or EEG.

7 We note that this interpretation is slightly different from the one that we offered in our previous ERP study which used a similar design but a different set of stimuli (Paczynski & Kuperberg, 2011, Experiment 1). In that study, we also reported a smaller negativity to animate than inanimate anomalies in the N400 time window, and we also attributed this difference to the fact that the animate but not the inanimate anomalies violated expectations based on the animacy hierarchy. In that study, we suggested that this led to reduced semantic processing of the anomalous animate (versus anomalous inanimate) nouns on the N400 component itself. However, because in the present investigation, Study 1 showed that the amplitude of the MEG N400 was just as large to the anomalous animate as to the anomalous inanimate nouns, we think that a component overlap explanation for the attenuation of the ERP N400 to the animate anomalous nouns is more likely.

8 In the earlier 300-500ms time window, the smaller EEG spatial similarity values for the anomalous animate (versus anomalous inanimate) nouns can be explained by the smaller evoked response to these animate continuations, as a result of an early start of the P600 component, as discussed above.

9 In predictive coding, “prediction error” is computed as an inherent component of the inference algorithm – the process of inferring semantic features from a word’s linguistic form. This differs from how prediction error has been simulated in other computational models of the N400, where the error is computed outside the model’s architecture (Fitz & Chang, 2019; Rabovsky & McRae, 2014). It also differs from computational models that have simulated the N400 as the total activity (Cheyette & Plaut, 2017; Laszlo & Plaut, 2012) or the total change in activity produced by a single set of units (Brouwer et al., 2017; Rabovsky et al., 2018; see Nour Eddine et al., 2022 for discussion and review). We also note that, although in the present study “prediction error” was produced by a linguistic “error” (an anomalous input), a linguistic error is not necessary to produce prediction error (or an N400) within a predictive coding framework. According to this framework, prediction error is produced at the lexico-semantic level whenever incoming lexico-semantic information cannot be explained by prior lexico-semantic predictions. This will result in a larger N400 to unexpected but plausible words, compared to expected words. We have recently argued that, in the case of anomalous inputs, prediction error is additionally generated at a higher event level of representation because the newly inferred anomalous event cannot be explained by still higher-level predictions based on longer-term real-world knowledge (Wang et al., 2023).

10 Note, however, that unlike the P300, which is thought to index the accumulation of raw evidence to make an initial decision, we have argued that the P600 reflects the accumulation of evidence that is gathered after the initial disruption in comprehension. In this sense, this component may bear more resemblance to later positivities within the P300 family of ERP components (Boldt & Yeung, 2015; Desender et al., 2019; Desender et al., 2021; Murphy et al., 2015; Steinhauser & Yeung, 2010), which are also thought to track the brain’s confidence in relation to a previous choice (Boldt & Yeung, 2015; Desender et al., 2019; Murphy et al., 2015; see Desender et al., 2021 for a review).

Additional information

Funding

This work was supported by National Institutes of Health: [Grant Number R01HD082527].