Neuroscience as a window into discourse processing: the influence of Walter Kintsch

ABSTRACT

Neuroscientific methods have greatly enhanced our understanding of cognition and language processing. However, their results are not always in line with predictions from models of discourse comprehension. In this essay I reflect on the impact Walter Kintsch’s work has had on neuroscientific investigations and on how models of discourse comprehension need to accommodate the blurring of distinctions that were previously taken for granted, and additionally, I sketch out a few of the recent developments in the field.

Neuroimaging as a window into discourse processing

When I started my graduate studies at the University of Colorado in Boulder in 1988, neuroscientists were only starting to develop methods to observe brain functions online. Neuropsychology and neurolinguistics were research endeavors medical departments were concerned with but that cognitive science could do well without. A few years later, toward the end of my doctoral studies, the field had greatly expanded; I even included one (!) reference to an ERP-study (event-related potentials) in my dissertation in 1994. I cannot remember if my advisor, Walter Kintsch, actually commented on that short paragraph at all—neuroscience was just not part of our toolbox yet. In this short period, positron emission tomography (PET) and functional magnetic resonance imaging (fMRI) had also come to fruition. To my knowledge, the very first—and still amazingly accurate—study on the neuroanatomy of text comprehension was published in 1993 (Mazoyer et al., 1993), but it was not yet obvious that neuroimaging would soon become widely available for the study of language processing.

The search for brain regions involved in text comprehension

The first ideas on the link between brain function and text comprehension came from the observation that some neurological patients without aphasic language deficits had considerable problems with communication at the text level. Symptoms such as incoherent language production, problems with inferencing, and difficulties with social and pragmatic communication had been described, often for patients with lesions in the right hemisphere. The right hemisphere hypothesis, thus, stated that word- and sentence-level language processing is mostly a left-sided affair, while including context and nonliteral language also involves both right and left hemispheres of the brain (Jung-Beeman, 2005). Neuroimaging studies did not readily confirm this idea, though (cf. Ferstl, 2007): Bilateral activation was observed even on the word level or sentence level (e.g., Mazoyer et al., 1993), and text comprehension did not always engage the right hemisphere. More-fine-grained empirical dissociations of subprocesses were needed. Obviously, the conceptual frameworks provided by Walter (Kintsch, 1988, 1998; Kintsch & van Dijk, 1978) were a good starting point.

Studies on cohesion and its consequences for coherence (Ferstl & von Cramon, 2001; Robertson et al., 2000), on inferencing (e.g., Ferstl & von Cramon, 2002; Friese et al., 2008; Kuperberg et al., 2006; Mason & Just, 2004; Virtue et al., 2008), and on situation model processing (Ferstl et al., 2005; Speer et al., 2007; Whitney et al., 2009) were conducted. The results converged on the key players: Besides the perisylvian language cortex, the activation patterns suggested an important role for the anterior temporal lobes, the left inferior frontal gyrus, and two midline regions—namely, the medial prefrontal cortex and the posterior cingulate cortex. “Fine,” Walter would have said, “Keep up the good work!” However, detailed inspection showed that prediction was close to impossible; which of these regions would show up in which fine-grained comparison remained a puzzle. The costs of fMRI prohibited the rigorous experimentation known from behavioral studies, varying any and all possible factors, such as task requirements, types of materials, imaging method, inference type, or participant characteristics. Attempts at such a program remained singular spotlights.

Nevertheless, I suggested some rather plausible mappings of Kintschian subprocesses of comprehension to brain regions (Ferstl, 2007; Ferstl et al., 2008). In particular, the anterior temporal lobes seemed to become engaged whenever words are combined to idea units or propositions (cf. Pylkkänen, 2020, for a recent account based on the linguistic concept composition). Interestingly, this neuroscientific finding might be interpreted as evidence for the textbase representation, in contrast to two-stage models of text comprehension that were partly based on neuroscientific observations of direct, embodied resonance (cf. Zwaan, 2004). Easy bridging inferences require processing of semantic information, as in the left inferior frontal lobe, and more strategic, explicit inferences involve the frontomedial cortex. However, other researchers developed quite different but, given the sparse data base, equally plausible interpretations. For instance, Mason and Just (2006) agreed on the function of the IFG for inferencing but saw the anterior temporal lobe (aTL) as important for text integration (rather than construction) and introduced a protagonist’s perspective interpreter for explaining the unexpected importance of the medial prefrontal cortex during narrative comprehension. Hagoort (2013) postulated unification as the process of “deriving new and complex meaning from the lexical building blocks” (p. 1)—a definition reminiscent of propositionalization or composition—as the function of the left IFG rather than the aTL. Finally, Egidi and Caramazza (2016) studied what they named accumulation and evaluation using an inconsistency paradigm (cf. Albrecht & O’Brien, 1993) very similar to the ones used in studies informed by Kintsch’s theory (cf. Ferstl et al., 2005; Helder et al., 2017).

Going beyond cognition

While the search for regions implementing Kintsch’s theory did not seem to be too successful, brain imaging confirmed what he taught his students: Language comprehension is not a modular, specialized function, but it is embedded in the cognitive system in a holistic way. Text comprehension without cognition—in particular, working memory, background knowledge, attention, and reading goals—is inconceivable (Kintsch, 1998). Consequently, the results from imaging studies yielded activations not only in what had been termed the language network—that is, Broca’s and Wernicke’s areas (cf. Fedorenko & Thompson-Schill, 2014; Lipkin et al., 2002)—but in the entire brain. Depending on the task and on the particular comparison, regions known to play a role in these and related cognitive processes were shown to be involved as well.

More importantly, though, our participants’ brains surprised us every now and again, as I have described previously (Ferstl, 2015). One region whose purpose was not readily mapped to one of the text comprehension subprocesses was the aforementioned medial prefrontal cortex. Ventromedial activations, usually related to reward and emotions in social interactions, were reported by Maguire et al. (1999), while the dorsomedial activations in inference studies (cf. Ferstl, 2015) had been implicated in the so-called theory-of-mind processes (Ferstl & von Cramon, 2002; Mar, 2011), or the processing of self-relevant information. Such noncognitive domains had been targeted in few studies in connection with text comprehension—for example, to elucidate the role of motivation or mood for successful comprehension or the representation of emotion—but they had not been explicitly included in theories of comprehension yet. The resulting paradigm shift from a purely cognitive theory to a more holistic view of comprehension as an integral part of and means for social communication was the most important consequence of neuroimaging studies of discourse processing.

Recent developments

The above summary of the neuroimaging literature on text comprehension has focused on the decade between 2000 and 2010. This period was the heyday of neuroimaging. The method was exciting, and each experiment yielded rich data. We gained truly novel insights that we happily communicated at specialized events, such as a conference at the Hanse-Institute for Advanced Studies in Delmenhorst in 2003 (organized by Chuck Perfetti and Franz Schmalhofer; Schmalhofer & Perfetti, 2007) and a week-long workshop at the Lorentz-Institute in Leiden (organized by Paul van den Broek, Jos van Berkum, and myself). In the last decade, however, rather than seeing an explosion of the field, neuroimaging has moved away from basic text comprehension research, a fate shared by other topics in classical experimental psychology. The replication crisis (cf. Zwaan et al., 2018), which hit neuroscience particularly hard (Nieuwland et al., 2018; Poldrack et al., 2017), and the shift from localization of specific brain regions to a network approach has made it less promising to conduct fine-grained, psycholinguistic experiments in the scanner. On the other hand, recent technological developments enable researchers to employ big-data approaches combined with bottom-up analyses to uncover patterns of brain function without the need for model-based hypothesis testing. This attractive and exciting paradigm shift seems promising even if it comes at the expense of text linguistic theoretical specificity.

Walter Kintsch’s lasting impact

Walter Kintsch would have embraced these trends, I am sure, despite recent doubts about the replicability and stringency of systemic neuroimaging research. When I once complained that his CI-model had too many degrees of freedom and could explain pretty much anything, Walter shrugged it off and told me he was more interested in the proof of concept rather than in falsifiability. He was always excited about novel possibilities and insights and focused on the big picture as much as on methodological issues. In this sense, his work was always ahead of its time. Although ChatGPT itself is not aware of its roots (see the funny query by Janice Keenan, this volume), Walter Kintsch’s work on latent semantic analysis (e.g., Foltz et al., 1998) helped to lay the groundwork for large language models (LLMs). He always went forward in his thinking. When, as a first-year student, I struggled with how the construction-integration model (Kintsch, 1988) fit with the distinction between micro- and macrostructure (Kintsch & van Dijk, 1978) he gave some quick response, but I could tell he found the question unproductive and a bit backward tending. When challenged in a seminar discussion about a detail in his work, he just shrugged it off: “Scientific theories merely are a proposal, a suggestion, based on what we know at a given point in time. It is the job of the next generation of researchers to take them apart and to come up with something better.” Neuroscience was a research field Walter happily left to his students and colleagues, but his thinking will live on in the work of the future generations.

Disclosure statement

No potential conflict of interest was reported by the author.