Comprehension Effects of Connectives Across Texts, Readers, and Coherence Relations

ABSTRACT

Studies investigating the effect of connectives on comprehension have yielded different results, most likely because of differences in methodology and limited samples of texts and readers. We added and removed causal, temporal, contrastive, and additive connectives in 20 authentic Dutch texts. Dutch adolescents (n = 794) differing in reading proficiency filled out four “HyTeC” cloze tests. Connectives were found to affect comprehension on a local level but not on a global level. However, a post-hoc analysis revealed a global comprehension effect for difficult texts but not for easy texts. Local effects were predominantly carried by the difficult texts as well. The direction of the effect did not vary between reading proficiency or readers’ educational level but did vary between types of coherence relations. Contrastive and causal connectives increased comprehension, whereas additive connectives reduced comprehension. Our large-scale study shows that effects of connectives on text comprehension are consistent between readers but not between texts and types of coherence relations.

Introduction

Do connectives such as “because” or “but” facilitate text comprehension? Despite extensive research on coherence and coherence markers, we still cannot confidently answer this question. Some studies show that coherence markers lead to better answers on comprehension questions (Sanders, Land, & Mulder, 2007) and better performance on probe recognition tasks (Millis & Just, 1994); other studies find no effect on free recall (Meyer, 1975; Sanders & Noordman, 2000) or on verification statements (Sanders & Noordman, 2000). Some authors have even claimed that connectives can negatively affect text comprehension (McNamara, Kintsch, Butler Songer, & Kintsch, 1996; Millis, Graesser, & Haberlandt, 1993). These offline comprehension findings differ from findings related to text processing, where connectives have consistently been found to benefit the integration of upcoming discourse segments (Cain & Nash, 2011; Cozijn, Noordman, & Vonk, 2011; Maury & Teisserenc, 2005; Sanders & Noordman, 2000; Van Silfhout, Evers-Vermeul, & Sanders, 2015; among others). Using larger numbers of texts and readers than in earlier work, we re-examine the important question to what extent connectives affect text comprehension while differentiating between coherence relations.

Coherence marking and comprehension

To understand a text, readers have to create a coherent mental representation in which text segments are related to each other. Connectives (and other lexical signals) can make such relations explicit, at a local level and at a global level (Pander Maat & Sanders, 2006).¹ They signal both the existence of a relation and the type of relation (such as causal, temporal, or contrast). As they reduce the inferential work left to readers, connectives should help readers create a coherent mental representation of the text.

However, such a beneficial comprehension effect has not consistently been found. Connectives can facilitate comprehension (Degand, Lefèvre, & Bestgen, 1999; Degand & Sanders, 2002; Loman & Mayer, 1983; Millis & Just, 1994; Sanders et al., 2007), but most comprehension studies show no effect of connectives or cohesive devices (Murray, 1995; Sanders & Noordman, 2000) or report mixed results, with facilitation occurring only in certain cases and not in others (Kamalski, Sanders, & Lentz, 2008; McNamara, 2001; McNamara et al., 1996; O’Reilly & McNamara, 2007; Ozuru, Dempsey, & McNamara, 2009; Spyridakis & Standal, 1987; Van Silfhout, Evers-Vermeul, & Sanders, 2014). Even when cohesion is increased by adding connectives in combination with other cohesive devices—such as making referential chains more explicit, eliminating ellipses, adding elaborative content and/or adding advanced organizers—increasing cohesion does not always result in increased comprehension. For instance, Freebody and Anderson (1983) increased cohesion by strengthening cohesive ties (argument overlap) and by adding connectives and cue phrases to their text fragments. However, they did not find an effect of cohesion on recall, summarization, or verification. More recently, Van Silfhout and colleagues (2015) found that adding connectives to history texts increased comprehension scores of prevocational and preuniversity 8th-grade students, but only for inference questions that directly tapped into the understanding of coherence relations and not for verification statements. They also found facilitation of connectives on situation-model questions (sorting task and timeline task) in one experiment but were unable to replicate these results in a second experiment.

Reasons for inconsistent comprehension results

Various factors may explain the discrepancies between studies investigating the effect of coherence marking on comprehension (see also Spyridakis, 1989a, 1989b; Degand & Sanders, 2002; Sanders et al., 2007). First, texts vary both in their content and stylistic difficulty across studies. The texts under investigation will determine which coherence relations are present to begin with and how conceptually difficult these relations are. The choice of texts used may therefore strongly influence the results, especially if the number of texts under investigation is low. For instance, Linderholm et al. (2000) increased the cohesion in a difficult and in an easy text (grade 12 vs. grade 9 on the Fry reading scale). They found that effects of coherence marking only increased comprehension for their difficult text and not for their easy text. Similarly, Spyridakis and Standal (1987) only found effects of connectives for their medium and high level texts (grades 11–16; Berta-Max Reading Level Analysis Program) and not for their low level text (grade 9). Effects of coherence marking thus strongly depend on the text under investigation, with a tendency to disappear in easier texts.

Second, individual differences between readers may lead to differences in the strength and the direction of comprehension effects. Readers with more prior knowledge benefit less from a highly cohesive text than from a low cohesive text, presumably because the latter forces them to actively engage and to process the text on a deeper level. Conversely, readers with low levels of prior knowledge benefit most from a highly cohesive text (Kamalski et al., 2008; McNamara, 2001; McNamara et al., 1996). Reading skills have also been found to interact with coherence marking effects. Ozuru et al. (2009) and O’Reilly and McNamara (2007) found that readers with high reading proficiency benefit more from coherence marking than readers with low reading proficiency (based on the Nelson-Denny reading test). In contrast, Geva and Ryan (1985) found that average and below-average readers from grades 5 and 7 (based on the Gates-MacGinitie Reading Comprehension Test) benefited more from coherence markers than above-average readers, even though they also found that low level readers had less knowledge of connectives (cf. Crosson & Lesaux, 2013). However, Geva and Ryan used relatively easy texts (grades 3 and 4) and note the possibility of ceiling effects for their above-average readers. This denotes the delicate balance between reading proficiency on the one hand and text difficulty on the other when investigating effects of coherence marking.

A third reason for the diverging results is that comprehension has been measured using a large variety of tasks (e.g., free, cued, or prompted recall; verification statements; bridging inference questions; situation-model questions; cloze; and summarization). Not all measures are equally sensitive to connective manipulations. Connectives tend to have a local impact: connecting adjacent clauses or sentences (Graesser, McNamara, & Louwerse, 2003). Methods like free recall may be too global to measure the local effects of coherence markers. The most consistent results are found when comprehension questions specifically target the content of manipulated relations (Sanders et al., 2007). However, even similar methodologies are difficult to compare. Situation-model questions are particularly difficult to compare across studies and texts. These questions are completely custom-made for a particular text, and the text will determine whether timeline, sorting, or schematic situation-model questions are most appropriate.

A final reason why coherence marking effects have differed across studies is that studies differ in the type of coherence relations they mark as well as the number of different signals and type of signals they use to mark them. Mixed-signal studies like Beck, McKeown, Sinatra, and Loxterman (1991) and McNamara et al. (1996) use various types of cohesive devices simultaneously: They added advanced organizers, headings, and connectives to increase both global and local coherence as much as possible (see also Freebody & Anderson, 1983; Loman & Mayer, 1983; Meyer, 1975; Ozuru et al., 2009; Vidal-Abarca & Sanjose, 1998). Other studies like Van Silfhout et al. (2015) have limited their experiments to the manipulation of local, adjacent relations with only connectives and cue phrases but did not discriminate between different types of coherence relations. As we discuss in the next section, there are strong indications that coherence markers will not affect the comprehension of different types of coherence relations in the same way.

Marking effects and types of coherence relations

On the basis of classical taxonomies of coherence relations (Halliday & Hasan, 1976; Sanders, Spooren, & Noordman, 1992) we focus on four main categories: additive (1), temporal (2), contrastive (3), and causal relations (4).

1. John went hiking and Bill went to the cinema.

2. John cooked supper. Afterwards, Bill did the dishes.

3. John likes to go fishing, but Bill hates the outdoors.

4. Bill cleaned up the kitchen because John left a mess.

There are strong reasons to expect that the effect of a connective will differ depending on the type of relation it marks. For one, multiple studies have found that contrastive relations are more difficult to understand and process than other relations (Golding, Millis, Hauselt, & Sego, 1995; Goldman & Murray, 1992; Haberlandt, 1982; Millis & Just, 1994). These findings fit in with a cognitive complexity approach: Types of coherence relations vary systematically in their relative cognitive complexity (Sanders et al., 1992). Young children acquire simple relations first, followed by increasingly complex relations. For instance, children produce positive relations (relations that are in line with expectations) before contrastive relations (relations that are contrary to expectation “not X”), and additive relations before causals (Evers-Vermeul & Sanders, 2009; Spooren & Sanders, 2008). In addition, complex relations require more time to process. Subjective relations, for instance, are processed slower than objective relations (Canestrelli, Mak, & Sanders, 2013; Traxler, Bybee, & Pickering, 1997).

From a cognitive standpoint, additive relations are the simplest relations. Truth-conditionally speaking, the additive relation is already implied by the juxtaposition of two text segments. Moreover, additive transitions offer minimal information on how the text might continue. Temporal relations introduce a temporal ordering and are therefore slightly more difficult. Contrastive and causal relations are considered more complex and stronger relations. Although contrastive and causal relations might be cognitively more complex than temporal or additive relations, they do tend to be better represented in memory. Meyer and Freedle (1984) found that texts with causal or contrastive structures are recalled better than texts with additive structures. Similar results have been found by Mulder (2008) on verification statements and by Sanders and Noordman (2000) on recall and verification statements.

We might ask whether coherence marking interacts with this complexity difference, in that weak, low-complexity relations may benefit less from marking than strong, high-complexity relations. After all, connectives marking complex relations typically provide more information on the relation than additive connectives such as “and” and “furthermore”. In a completion task Koornneef and Sanders (2013) found that an additive connective elicited a much wider range of coherence relations than a causal or contrastive connective. Therefore, additive relations might not supply very clear processing instructions, compared with other connectives. However, Sanders and Noordman (2000) found no interaction between coherence marking and coherence type (additive [list] vs. causal [problem-solution]) on reading times, recall, or verification statements. Murray (1995) found mixed results. In his first experiment he found an interaction between coherence type and coherence marking. Introducing an appropriate connective increased cued recall for causal relations but decreased recall for additive relations. No effect was found for contrastive relations. However, this pattern was not replicated in Murray’s second experiment.

Another factor influencing comprehension effects of coherence marking might be the continuous or discontinuous nature of the transition involved. Murray (1997) proposed that coherence relations are marked according to the “Continuity principle.” By default, readers expect consecutive sentences to describe events that follow each other in a continuous manner. If the events are discontinuous and there is no signal preparing the reader for a disruption in the continuity of events, comprehending the sequence is more difficult.² According to the Continuity principle, connectives will benefit comprehension and processing the most when they mark a discontinuous relation. This includes connectives that mark a nonlinear order of events (e.g., “because” or “before”). Even additive relations might not be completely continuous. The second segment can elaborate on the first—in which case it continues the topical focus of the prior segment—but the segments may also be parallel as in a list. In a list, the second segment introduces a new point that refers back to a higher level topic or referent (Knott, Oberlander, O’Donnell, & Mellish, 2001; Pander Maat, 2001, 2002).³ In this sense it does not flow from the first segment as smoothly as elaborations do. Similarly, although contrastive connectives signal discontinuity, not all contrastive connectives are equally discontinuous. The connective “but” attributes focus to the segment following the connective (S2) and tends to be used when the focus in the next sentence also lies on S2, whereas “although” in S2 indicates that the focus shifts back to the S1 in the next sentence (McClure & Geva, 1983).

Additional evidence for differences in marking effects between types of coherence relations comes from corpus research. Asr and Demberg (2012) studied the frequency distribution of relations in the Penn Discourse Tree Bank corpus (Prasad et al., 2008). The frequency with which a relation was explicitly marked differed between coherence types. Drawing on the Continuity hypothesis and the Causality-by-Default hypothesis (Sanders, 2005), they hypothesized that causal relations and relations that did not disrupt the continuity of the text (i.e., noncontrastive and in forward order) would be explicitly marked less often than other relations.⁴ Their findings showed support for both hypotheses. Thus, it seems likely that marking continuous relations is less effective than marking discontinuous relations.

In sum, evidence from acquisition, processing, and corpus studies suggests that the presence of a coherence marker may not affect different types of coherence relations in the same way. More complex and less continuous relations may depend more strongly on explicit marking than other types of relations to be correctly understood. Based on the Cognitive complexity account and the Continuity principle, we hypothesize that of all relations contrastive relations should benefit most strongly from coherence marking, followed by causal, temporal, and additive relations.

Present study

We have suggested that the inconsistent effects of coherence marking on comprehension in earlier work are likely caused by differences in methodology, limited samples of texts and readers, and generalizations across cohesive devises and types of coherence relations. In our study we aim to control these factors. We examine whether 20 authentic Dutch texts benefit from adding only additive, temporal, causal, and contrastive connectives. In contrast to previous studies, we use a large number of texts that were randomly selected from a larger corpus. The texts differ both in the number of coherence relations and in the types of coherence relations they contain. The texts are presented to readers varying in reading proficiency to see whether effects of connectives can be generalized over texts and readers.

Methods

This study was part of a larger project in which the difficulty of 60 texts was assessed among 2,926 participants. Texts and participants were randomly assigned to one of three experimental substudies, each of which investigated the effect of a linguistic feature on text comprehension using the same methodology and design (see Kleijn, 2018; in addition to the current cohesive device manipulation, the project manipulated texts for syntactic and lexical complexity).

Participants

Thirty-five Dutch secondary schools participated with a total of 794 students from grades 8, 9, and 10 (ages 13–16 years). Testing was introduced as part of the regular school curriculum. The students were enrolled in different levels of secondary education: in preuniversity education (“vwo”), general secondary education (“havo”), or prevocational education (“vmbo-gt”, “vmbo-kb” or “vmbo-bb”).⁵ Table 1 shows the distribution of participants over grades and education levels. Grade 10 students of prevocational education were not included in the study because prevocational students graduate after grade 10 and were busy preparing for their final exams.

Table 1. Distribution of Participants over Grade and Education Level

	Education Level
Grade	Prevocational (Low)	Prevocational (Medium)	Prevocational (High)	General Education	Preuniversity	Total
Grade 8	24	72	175	54	45	370
Grade 9	34	50	96	63	71	314
Grade 10	-	-	-	75	35	110
Total	58	122	271	192	151	794

Materials

The materials for this study were selected as part of a larger project (see Methods). The project used an elaborate quasi-random statistical selection procedure (adapted from Liu, Kemper, & Bovaird, 2009) to select a diverse sample of 60 texts from a collection of authentic Dutch texts and to distribute these texts randomly over three experimental substudies (for the complete procedure see Kleijn, 2018, pp. 10–12). Following this procedure, 10 educational textbook texts and 10 public information texts were assigned to the present study. The educational texts came from textbooks on history, geography, Dutch language, and economics. These texts were written specifically for students in secondary education. The public information texts were written for the general public but dealt with topics judged relevant to Dutch adolescents: health (e.g., donor registration), the environment (e.g., pest control), and specific regulations (e.g., obtaining a scooter/moped license). Texts were 300 to 410 words long and did not contain figures or tables.

We were careful to use natural text to create experimental versions that differed in only one factor: linguistic marking of the relations. To identify all coherence relations in the text, explicit and implicit relations were analyzed, following a text-analytical procedure (Pander Maat, 2002). Two text versions were created by adding or removing causal, contrastive, temporal, or additive connectives. In the “low-coherence marking text version” (Low-CM), some relations were left implicit, whereas in the “high-coherence marking text version” (High-CM) the same relations were explicitly marked with a connective.⁶ Only one-third of the coherence relations present in the text were manipulated⁷ to keep the text natural. As a result, Low-CM text versions still contained connectives and other cohesive devices, and not all relations in High-CM text versions were explicitly marked (see Appendix A). We only added connectives that were relatively frequent and are familiar to our participants.⁸ Most of our manipulated connectives had a local impact: marking coherence relations between consecutive clauses and sentences. Connectives were only left out if they were optional, and leaving the connective out did not alter the interpretation of the relation or the flow of the text. The text analysis ensured that connectives that were added to the relations were appropriate (cf., Cain & Nash, 2011; Millis et al., 1993; Murray, 1997) and that manipulations did not change the meaning of the relation. Text versions were kept as close to the original as possible. Because we only adapted coherence relations that were already present in the original texts, it depended on the text at hand which types of coherence relations were manipulated.

In total, 193 relations were manipulated (Table 2). Appendix B shows which types of coherence relations were manipulated in each text. Half of all the manipulated relations were causal (5), including objective as well as subjective causal relations in forward and in backward order (e.g., Pander Maat & Sanders, 2006). There were only a few temporal relations (6). Contrastive relations (7) and additive relations (8) were well represented in the sample. Additive relations included both list and elaboration relations.

• (5) {Causal relation} If the production structure improves, the country can better compete with other countries on the global market. As a result, the county’s export can increase and the country’s import can decrease.

• (6) {Temporal relation} In the late Middle ages, we find more and more cities in Europe. Tradesmen formed trading communities at locations with rich customers, like a noble house, a military stronghold or a monastery. Subsequently, these communities attracted craftsmen.

• (7) {Contrastive relation} Minors can record their preference in the Donor Registers from age twelve. Parents or guardians do not have to provide consent for this. But if minors agree to become a donor and die before they are sixteen, parents or guardians can still refuse.

• (8) {Additive relation} There are a lot of consequences when the fire department is notified by a fire alarm. First of all, the fire department rushes to the location of the alarm. This can result in dangerous traffic situations. Secondly, if the fire department responds to a false alarm, it will not be available to provide aid in a genuine rescue situation.

Table 2. Frequencies of Manipulated Coherence Relations per Coherence Type

Coherence Relation	Frequency
Causal	100
Contrast	42
Additive	42
Temporal	9
Total	193

Measures

Comprehension assessment

Text comprehension was measured using Hybrid Text Comprehension cloze tests (HyTeC-cloze tests; Kleijn, Pander Maat, & Sanders, 2019). The HyTeC-cloze was developed as an alternative to the standard cloze and other standard comprehension assessment measures, using insights from discourse representation studies, readability research, and language assessment research. The HyTeC-cloze combines the strengths of mechanical and rational cloze tests into a valid and reliable measure of text comprehension.⁹ It enables us to directly compare coherence marking effects across texts, since performance depends on text difficulty without interference from question difficulty (Klare, 1976). Furthermore, the HyTeC-cloze can be used to simultaneously investigate comprehension effects on a global text level and on a local level (e.g., targeting specific manipulations). HyTeC-cloze tests are produced in three steps:

Step 1: selection of possible gap candidates following a rational strategy

A rational strategy is used to prevent words that do not rely on text level comprehension from becoming cloze gaps. This includes words that can be reconstructed using only grammatical knowledge or knowledge of usage conventions (e.g., articles, multiword expressions). Also excluded are words that can only be guessed, such as names and numbers. Excluding these words as potential cloze gaps makes the cloze test more sensitive to text comprehension, since scores are not affected by answers that can be provided with only the immediate context or gaps that are practically impossible to fill out correctly (regardless of how much context is taken into account). To preserve our coherence marking manipulation, inserted connectives were also prevented from becoming cloze gaps. All other words in the text were candidates for deletion.

Step 2: distribution of gap candidates over different cloze versions following a mechanical strategy

Gap candidates were divided over different cloze versions via a mechanical strategy, which ensured a random sample of candidates and an equal distribution of gaps throughout the text. The HyTeC-procedure uses a deletion ratio of 1 in 10: for a 300-word text 30 gaps are selected. If the text has 120 gap candidates, four unique cloze tests can be created that each sample a different set of words; cloze gap candidates 1, 5, and 9 end up in version 1; candidates 2, 6, and 10 end up in version 2; and so on. As a result, gaps were present in sentences to which a connective was added or removed (“host sentences”) and in sentences that were not manipulated.

Step 3: random selection of two cloze versions out of all possible versions

Finally, two of the cloze versions were randomly selected to serve in the study. Both cloze versions were used in the Low-CM and High-CM condition, resulting in four different versions of each text: cloze version 1/Low-CM, cloze version 1/High-CM, cloze version 2/Low-CM, cloze version 2/High-CM. Cloze gaps in the Low-CM and High-CM versions were identical.

Reading proficiency

Standardized reading proficiency scores were collected for all students. Students either participated in the “Student monitoring system secondary education” (VVO) battery test from Cito (“The Dutch institute for Educational Measurement”), which includes a reading proficiency test, or were given a reading proficiency test from the test battery RSM14, also developed by Cito. Both tests are written multiple choice tests that measure the ability to understand written texts and to reflect on text function and goals.¹⁰ Although scores of both tests were mapped to the same scale, analyses showed that scores were consistently higher for one of the tests. To control for this, the factor Reading test was included in the analyses.

Design

The experiment was set up following a matrix sampling design (e.g., Gonzalez & Rutkowski, 2010). Each participant was given four different cloze tests: one cloze test of a Low-CM educational text, one of a Low-CM public information text, one of a High-CM educational text, and one of a High-CM public information text. To balance out possible order effects, each combination of cloze tests was also presented in reversed order.

Procedure

All testing took place at the participating schools. The cloze tests were administered by the teachers in classroom settings and were presented on computers. Participants completed their four cloze tests in two 45-minute sessions. Schools scheduled all sessions themselves over the course of several weeks.

Scoring procedure and data cleanup

The answers to each cloze gap were dichotomously scored (1 = correct; 0 = incorrect) according to the acceptable word scoring procedure (Oller & Jonz, 1994). This means that not just originally deleted words but also semantically equivalent alternatives were scored as correct (while also being lenient toward spelling errors and typos). The acceptability of alternative answers was judged by the global appropriateness criterion: The answer had to fulfill “all the contextual requirements of the entire discourse context in which it appears” (Oller & Jonz, 1994, p. 416). Each answer was scored by 2 independent judges from a pool of 16 judges. When the judges disagreed, a third judge made the final decision. All judges took part in a training session to familiarize them with the scoring procedure. The judges agreed in 86% of the cases. The internal reliability of the cloze tests was also calculated and was found to be reliable (k = 40; mean Cronbach’s alpha = .809; SD = .036; range, .707–.869; see also Kleijn et al., 2019).

Ten percent of the cloze tests were removed because students repeatedly gave nonserious answers or did not provide an answer on any of the cloze gaps. Cases where students occasionally failed to fill in a gap were regarded as incorrect answers rather than missing answers. The final dataset contained 99,735 cases within 2,861 cloze tests.

Analyses

The data were analyzed at the response level using generalized linear mixed-effect modeling with a logit link. Each case represented an answer to an individual cloze gap. Observations were nested within students and texts, with students nested in schools.

Two separate analyses were performed: a text level analysis and a relation level analysis. The goal of the first analysis was to see if increasing the number of connectives within a text influences overall cloze performance. All cloze gaps were included in this analysis (i.e., cloze gaps in manipulated sentences as well as in unmanipulated sentences). The students’ reading proficiency, grade, and education level were included to test for possible interactions between the coherence marking manipulation and reader characteristics. Finally, the text feature Genre (educational textbook or public information) was included in the analysis to test the generalizability of effects over genres.

Given that most effects in prior research have been found on questions that targeted the relation itself, it was likely that effects of the manipulation were restricted to the segments surrounding the connective and would not affect performance on all cloze gaps. Therefore, we ran the second analysis using only cloze gaps from sentences in which a connective was added or removed (“host sentences”). Although the impact of connectives might exceed the host sentence, we expect any effect of coherence marking to be the strongest here.

A second goal of the relation level analysis was to test whether the effect of coherence marking is moderated by the type of coherence relation. Adding a contrastive connective, for instance, may not have the same effect as adding an additive connective. This hypothesis was explored by including a 2-way interaction of Coherence Marking with Coherence Type in the analysis. We expect the strongest facilitation for contrastive relations and slightly less facilitation for causal relations since causal relations are generally more expected than contrastive relations. We expect additive connectives to have either a very small positive effect on text comprehension or an adverse effect. On the one hand we hypothesize that additive relations will have a smaller facilitation effect than contrastive and causal relations, since additive relations are a continuation of the text (and thus expected). Furthermore, additive connectives are not very informative regarding the content of the upcoming segment. On the other hand, because such continuations tend to remain implicit, marking the relation may have an adverse effect. Highlighting a relation that is usually expressed without a connective could disrupt comprehension rather than facilitate it. Descriptions of all factors included in the analyses are given in Table 3.

Table 3. Descriptions of Factors

Factor	Description	Levels
Coherence marking	Text version: high or low coherence marking	2 levels
Education level	Level of education in which the student is enrolled	5 levels
Grade	Grade in which the student is enrolled	3 levels
Reading proficiency	Reading proficiency score on Dutch reading ability test (centered and standardized)	Continuous
Reading test	Reading test used to test reading ability	2 levels
Genre	Educational text or public information text	2 levels
Coherence type	Type of manipulated coherence relation	4 levels

Results

Text level analysis

Table 4 shows the mean percentage of gaps answered correctly per text version, Education level, and Grade. The analysis revealed main effects for Education level, Grade, Reading proficiency, Reading test, and Genre but not for Coherence marking. The odds of a correct answer increased with Education level, Grade, and Reading proficiency score. Performance on educational texts was better than on public information texts. The model including Coherence marking is presented in Table 5.

Table 4. Mean Probability Correct per Text Version, Education Level, and Grade

		Coherence Marking
Education Level	Grade	Low-CM	High-CM
Prevocational (low)	Grade 8	30.04%	37.80%
	Grade 9	34.75%	33.86%
Prevocational (medium)	Grade 8	42.06%	40.40%
	Grade 9	48.39%	47.79%
Prevocational (high)	Grade 8	48.66%	48.60%
	Grade 9	53.40%	54.52%
General	Grade 8	55.43%	57.87%
	Grade 9	61.82%	61.26%
	Grade 10	63.96%	68.02%
Preuniversity	Grade 8	65.25%	67.24%
	Grade 9	69.66%	69.89%
	Grade 10	70.49%	72.67%

Table 5. Model Text Level Analysis (Including Nonsignificant Factor Coherence Marking)

	Estimates	SE	Z	p	Odds Ratio
Random effects
School	0.015	0.010	1.500	.067^a
School: student	0.212	0.013	16.308	<.001^a
Text	0.038	0.007	5.429	<.001^a
Fixed effects
Intercept	−0.409	0.125	−3.272	<.001	0.664
Coherence marking: High-CM	0^b
Coherence marking: Low-CM	−0.020	0.014	−1.429	.153	0.980
Education level: prevocational low	0^b
Education level: prevocational medium	0.324	0.093	3.484	<.001	1.383
Education level: prevocational high	0.578	0.093	6.215	<.001	1.782
Education level: general	0.801	0.112	7.152	<.001	2.228
Education level: preuniversity	0.864	0.125	6.912	<.001	2.373
Grade 8	0^b
Grade 9	0.109	0.045	2.422	.015	1.115
Grade 10^c	0.212	0.096	2.208	.027	1.236
Reading proficiency	0.308	0.029	10.621	<.001	1.361
Reading test: RSM14	0^b
Reading test: VVO	−0.535	0.091	−5.879	<.001	0.586
Genre: public information	0^b
Genre: educational textbook	0.343	0.026	13.192	<.001	1.409

^aOne-sided.

^bSet as reference level.

^cUnbalanced, no 10th grade prevocational students in the sample.

Relation level analysis

After the text level analysis, a relation level analysis was conducted to find out whether effects of coherence marking can be found in the manipulated sentences (i.e., host sentences). This analysis also enables us to investigate the possible interaction between linguistic marking of coherence relations and the type of coherence relation. Cloze gaps were selected from sentences to which a connective was added or from which a connective was removed. Five sentences were excluded from the analysis because they contained not one but two manipulated connectives that each marked different types of coherence relations (e.g., first connective marked relation with previous sentence; second connective marked relation between two clauses within the sentence); 183 items remained with a total of 38,214 observations. Table 6 shows the mean percentage of gaps that were answered correctly per text version, Education level, and Grade for the host sentences.

Table 6. Mean Probability Correct in Host Sentences per Text Version, Education Level, and Grade

		Coherence Marking
Education Level	Grade	Low-CM	High-CM
Prevocational (low)	Grade 8	28.49%	37.40%
	Grade 9	32.02%	32.45%
Prevocational (medium)	Grade 8	40.68%	39.44%
	Grade 9	45.83%	48.07%
Prevocational (high)	Grade 8	47.34%	48.12%
	Grade 9	51.35%	53.27%
General	Grade 8	53.56%	57.19%
	Grade 9	59.20%	59.90%
	Grade 10	63.05%	67.42%
Preuniversity	Grade 8	64.11%	67.96%
	Grade 9	69.03%	67.97%
	Grade 10	71.72%	71.52%

The analysis revealed main effects for Coherence Marking, Education level, Grade, Reading proficiency, Reading test, and Genre. Performance was better in the High-CM version and increased with Education level, Grade, and Reading proficiency. Odds of a correct answer were also higher for educational texts than for public information texts. Students performed better on High-CM host sentences than on Low-CM host sentences. However, when Coherence type was added to the model, the overall effect of Coherence marking disappeared, and a significant interaction between Coherence marking and Coherence type was found. This final model is presented in Table 7. Causal and High-CM are selected as reference levels for Coherence type and Coherence marking, respectively. The estimate given after “Low-CM” therefore denotes the estimated effect for causal relations when they are not marked with a connective. This effect is negative but only marginally significant. For the other coherence types the “Low-CM” estimate is adjusted and can be found by adding up the “Low-CM” estimate and the estimate for the interaction “Low-CM*Contrast,” “Low-CM*Temporal,” or “Low-CM*Additive,” respectively. The estimated effect of removing connectives in contrastive relations is therefore −0.060 + −0.147 = −0.207, for temporal −0.060 + −0.018 = −0.078, and for additive −0.060 + 0.165 = 0.105. For temporal relations, this adjustment is not significant, but for contrastive relations and additive relations it is. The coherence marking effect for contrastive relations is in the same direction as for causal relations (i.e., Low-CM is more difficult), but the effect for contrastive relations is significantly larger than for causal relations. Conversely, for additive relations the effect is also stronger than for causal relations but in the opposite direction. Additive relations were easier when they were not marked by a connective. In sum, coherence marking significantly improved cloze performance for contrastive relations and marginally improved performance for causal relations. It had no effect on temporal relations and a negative effect on additive relations (Figure 1). No interactions with Education level, Grade, Reading proficiency, or Genre were found.

Table 7. Final Model Relation Level Analysis

	Estimates	SE	Z	p	Odds Ratio
Random effects
School	0.032	0.013	2.462	<.001^a
School: student	0.189	0.015	12.600	<.001^a
Sentence	0.039	0.007	5.571	<.001^a
Fixed effects
Intercept	−0.284	0.140	−2.029	.042	0.753
Coherence marking: High-CM	0^b
Coherence marking: Low-CM	−0.060	0.031	−1.935	.053	0.942
Coherence type: causal	0^b
Coherence type: contrastive	−0.249	0.038	−6.553	<.001	0.780
Coherence type: temporal	−0.224	0.078	−2.872	.004	0.799
Coherence type: additive	−0.022	0.040	−0.550	.582	0.978
Low-CM * contrast	−0.147	0.054	−2.722	.006	0.863
Low-CM * temporal	−0.018	0.110	−0.164	.870	0.982
Low-CM * additive	0.165	0.057	2.895	.004	1.179
Education level: prevocational low	0^b
Education level: prevocational medium	0.356	0.101	3.525	<.001	1.428
Education level: prevocational high	0.578	0.102	5.667	<.001	1.782
Education level: general	0.802	0.122	6.574	<.001	2.230
Education level: preuniversity	0.903	0.137	6.591	<.001	2.467
Grade 8	0^b
Grade 9	0.099	0.048	2.063	.039	1.104
Grade 10^c	0.183	0.105	1.743	.081	1.201
Reading proficiency	0.296	0.031	9.548	<.001	1.344
Reading test: RSM14	0^b
Reading test: VVO	−0.541	0.104	−5.202	<.001	0.582
Genre: public information	0^b
Genre: educational textbook	0.141	0.025	5.640	<.001	1.151

^aOne-sided.

^bSet as reference level.

^cUnbalanced, no 10th grade prevocational students in the sample.

Figure 1. Final model estimated interaction effect Coherence marking and Coherence type. Estimates are set on the reference levels for Education level, Grade, Reading proficiency, Reading test, and Genre (see Table 7).

Figure A1. Average number of connectives per text and text version. Calculated over the whole text, including connectives that were not manipulated.

Figure A2. Distribution of manipulated relations across texts per coherence type.

Discussion

We investigated the effect of linguistic marking of coherence relations between local, adjacent sentences in 20 randomly selected authentic Dutch texts. Coherence marking was manipulated by adding or removing causal, temporal, contrastive, or additive connectives. Texts were presented to secondary education students differing in reading proficiency, grade, and education level. Their comprehension of the texts was measured using HyTeC-cloze tests. Our results suggest that adding connectives to a text tends to affect comprehension on a local level but not on a global text level. These results are in line with previous findings that show that connectives mainly function locally in texts (e.g., Canestrelli et al., 2013; Graesser et al., 2003; Sanders et al., 2007; Van Silfhout et al., 2015).

We also found that the direction of the comprehension effect depends on the type of coherence relation. Connectives facilitated comprehension of contrastive and causal relations but decreased cloze scores in additive relations. The results for contrastive and causal connectives are in line with our expectations. These relations represent the connections with the most constraints and as such are the most informative with regard to the upcoming segment. Readers benefitted most from the linguistic marking of contrastive relations. This effect occurred despite of the presence of other contrastive cues that were left intact to keep the sequence interpretable as a contrastive relation. Causal connectives only showed a trend toward facilitation. This may seem surprising given the relatively large amount of literature showing a facilitating effect of causal connectives. However, most of this literature shows an effect on online processing and is not focused on comprehension. One way to explain the limited effect of causal connectives in comprehension is that readers tend to make causal connections whenever they can (Myers, Shinjo, & Duffy, 1987; Sanders, 2005) and have a preference to connect segments causally. Adding a connective only confirms their expectation, which might explain the only marginally significant effect for causals. Another possibility is that the set of causal relations used in this study was too heterogeneous and that different subtypes have opposite or reduced effects. However, post-hoc analyses in which causal relations were subdivided by subjectivity and linear order did not show any difference between subtypes of causal relations.

Connectives signaling additive relations decreased comprehension. It seems readers were thrown off track rather than guided by these connectives. It may be that highlighting additive relations drew unnecessary focus toward the integration of the segment. Rather than taking two consecutive segments as sharing a (sub)topical focus, an interpretation invited by a simple juxtaposition, additive connectives may have invited readers to look at this relation more closely. This may be especially the case for list relations. For these relations, segments “A” and “B” are parallel elements connected under a global topic. Marking this relation will force the reader to identify this topic. It thereby blocks the interpretation that B is a continuation of A, elaborating on the same subtopic. In addition, it forces readers to identify the shared topic and reinterpret A as part of a list as well. This may force the reader to restructure his mental representation (see Kintsch & Vipond, 1979). In addition, because A and B are separate points, their relation offers little information regarding the content of either segment (compared with contrastive relations for instance, in which one of the segments contradicts an expectation based on the other segment). In elaboration relations, the elaboration is a natural continuation of the first segment and therefore does not require restructuring of the mental representation. An elaboration connective only confirms this default continuation of the text and hence adds little information on how the text will continue. Post-hoc analysis of the data did show a trend toward Coherence Marking only negatively affecting list relations and not affecting elaboration relations, but the analysis did not reach statistical significance (β = −0.171, SE = 0.108, p = .113). As such, these are speculations indicating the need for more theoretical and experimental research into different types of coherence relations and their linguistic expression.

Finally, for temporal relations adding a connective did not seem to benefit or hinder comprehension. However, we must note that temporal relations were under-represented in the materials (only seven cases in the relational analysis). The unbalanced distribution of connective types in the sample was a small drawback of choosing to use natural texts.

Because of the large numbers of texts, our findings are robust and do not rely on one or two experimental texts. Still, connective effects may differ between texts. Linderholm et al. (2000) and Spyridakis and Standal (1987) found that coherence marking did not affect easy texts. This raises the question whether coherence marking effects depend on text complexity in our data as well. Our texts were randomly selected from a wide range of texts, including easier and more difficult texts. A post-hoc exploration of the overall cloze scores of the texts showed mean scores ranging from 27% correct up to 75%. To see whether text complexity influenced the effects of coherence marking, we split the dataset in two subsamples of 10 texts each, based on the mean text scores. We conducted a clear split: Texts in the middle of the spectrum were included because the distribution of coherence types and genres across the splits did not differ. Text level analysis and relation level analysis were conducted separately for each subsample. At text level there was no effect of Coherence Marking for easy texts, but there was a significant positive effect for difficult texts (β = −0.067, SE = 0.022, p = .002). We checked whether this effect was carried by the cloze gaps in host sentences (sentences where connectives were added or removed) or whether it transcended the boundaries of these sentences. The Coherence marking effect for nonmanipulated sentences was significant as well (β = −0.071, SE = 0.028, p = .011). This finding indicates that for difficult text, connectives can have global comprehension effects.

At relation level, Coherence marking only affected comprehension of difficult texts. Difficult texts showed a pattern similar to the one found in the entire dataset: Contrastive connectives had a positive effect, causal connectives showed a slight positive trend, and additive connectives showed a negative effect on comprehension. Hence, it seems likely that the effects found in the original relation level analyses were predominantly carried by the difficult texts in the sample. More generally, these results support the idea that effects of coherence marking on comprehension tend to disappear in easy texts. When texts are easy enough, connectives have no added value for building a coherent mental representation of the text.

Our participant sample was drawn from secondary school students enrolled in different levels of the Dutch educational system. It included students from different language backgrounds, students with low reading proficiency and students with high reading proficiency. Contrary to findings of O’Reilly and McNamara (2007) and Ozuru et al. (2009), our coherence marking effect appeared independent of reading proficiency. Both high-proficiency and low-proficiency readers performed better when strong relations were marked with a connective. Coherence marking also did not interact with Education level. Students enrolled in the highest levels of Dutch secondary education benefitted as much as students enrolled in the lower levels.

The lack of an interaction effect between reader characteristics and coherence marking is especially interesting given the finding that coherence marking effects seemed to disappear in easy texts. We would then expect that poor readers would show a stronger effect of coherence marking than good readers, since for them “easy” texts are still more difficult than for proficient readers. However, we did not even find a trend in that direction. It suggests that it is not the reader’s skills that determine the difficulty level of the text, at least not when it comes to the effectiveness of connectives. We believe that easy texts discuss simple concepts in a clear manner. As a result, creating a coherent representation of the text is quite easy to do and there is less need for explicit processing signals like connectives.

The results above also confirm the validity and functionality of the HyTeC-cloze test in studies that investigate the effects of linguistic marking on comprehension. The HyTeC-cloze was sensitive to coherence marking and even detected differences between relation types. The test has multiple advances over more traditional methods, including fast application, no interference from question difficulty (cf. Klare, 1976), and the possibility to investigate global text level and local level comprehension effects simultaneously.

In conclusion, our results show that connectives can facilitate or disrupt comprehension depending on the coherence relation that they mark, particularly in demanding texts. Whether this holds for other coherence markers such as global coherence markers, cue phrases, or even verbs and syntax (Das & Taboada, 2017; Halliday & Hasan, 1976) is yet to be determined. By teasing apart different types of coherence relations, our study has provided more insight into the complex mechanisms that influence how coherence is established by the reader.

Disclosure statement

No potential conflict of interest was reported by the authors.

Additional information

Funding

This work was supported by NWO (‘The Netherlands Organisation for Scientific Research), Cito (“The Dutch Institute for Educational Measurement”), and De Nederlandse Taalunie (“The Dutch Language Union”) under NWO grant number 321 89 002.

Notes

1. Throughout this article we define “coherence” as a phenomenon that is located in the mind of the reader and “cohesion” as the linguistic expression of coherence (e.g., Sanders & Pander Maat, 2006).

2. Zwaan and colleagues have presented similar views on processing continuous event structures versus discontinuous event structures (Zwaan, Langston, & Graesser, 1995; Zwaan & Radvansky, 1998).

3. Additive relations may also share “joint relevance” where together the two segments answer one topic question (“What did you do this summer?” “I painted my house and visited family.”; Pander Maat, 2001).

4. Cf. Taboada (2009) on what constitutes an implicit relation.

5. These five levels are ordered from academically oriented education to practice-oriented education. For more information on the Dutch educational system see EP-Nuffic (2015).

6. We manipulated the original text in two directions. If a relation was explicitly marked, we kept the marker in the High-CM text version and removed the marker in the Low-CM version. If a relation was implicit, we kept the relation implicit in the Low-CM version and added a connective to the High-CM version.

7. If it was not possible to manipulate one-third of the coherence relations in a text, another text was randomly selected to take its place. If more manipulations were possible, the more complex relations were chosen.

8. Mean frequency of the added connectives was 5.47 (SD = 0.85; range, 3.79–7.30) occurrences per billion of words (log-transformed, based on the SoNaR-corpus; Oostdijk, Reynaert, Hoste, & Van Den Heuvel, 2013).

9. The HyTeC-cloze was found to match and on occasion outperform standardized tests of reading ability on validity and reliability tests: Internal reliability was on average .828; scores correlated .6 with standard reading ability and vocabulary tests, the tests was more sensitive to higher level constraints than standard cloze tests, and the test was sensitive to known-group differences between readers, texts, and text versions (for detailed analyses see Kleijn et al., 2019).

10. Questions target local and global processes, including comprehension of the main ideas of the texts; comprehension of words, sentences, paragraphs in context, and the relations that hold between them; and understanding differences between facts, opinions, claims, arguments, and conclusions. Cito-tests are rigorously tested for validity and reliability (Van Til & Van Boxtel, 2015). Internal reliability of the VVO and RSM14 reading proficiency tests is on average .75.

Appendix A.

Connectives per Text and Text Version

Figure A1 shows the number of connectives in the original text versus the number of connectives in the Low-CM and High-CM text versions. The number of connectives in each text was determined using T-Scan. The number was standardized to connectives per clause.

Appendix B.

Distribution of Manipulated Relations Across Texts