Automated and controlled processes in comprehending multiple documents

ABSTRACT

The study investigates automated and controlled cognitive processes that occur when university students read multiple documents (MDs). We examined data of 401 students dealing with two MD sets in a digital environment. Performance was assessed through several comprehension questions. Recorded log data gave indications about students’ time allocation, corroboration, and sourcing. Independent measures were used for reading speed to tap the effects of automatic processing and for working memory and single-text reading comprehension to tap effects of controlled processing, with working memory considered the mental capacity for performing controlled processing. We found that faster readers completed the MD tasks faster and showed more corroboration behavior. At the same time, students skilled in comprehension allocated more time to processing MD tasks and were more likely to show MD-specific behaviors of corroboration and sourcing. Students’ success in MD tasks was predicted by reading speed and working memory, with the effect of working memory being mediated by single-text comprehension. Behavioral indicators contributed independently in predicting students’ MD comprehension. Results suggest that reading MDs resembles a problem-solving situation where students need to engage in controlled, non-routine processing to build up a comprehensive representation of MDs and benefit from highly automated, lower-level reading processes.

KEYWORDS:

• Multiple documents
• reading speed
• working memory
• reading comprehension
• processing behavior

Introduction

Reading and learning from multiple documents and multiple sources is a typical task of higher education learning and academic work (Britt and Rouet 2012). Nevertheless, reading multiple documents (MDs) can be challenging beyond performing regular reading procedures. It requires students to compare, evaluate and integrate different text and meta-text information across documents and to represent information embedded in specific document contexts (e.g. Anmarkrud, Bråten, and Strømsø 2014; Mahlow et al. 2020; Schoor, Hahnel, Mahlow et al. 2020). In this regard, MD reading shares similarities with processes of problem-solving. Britt, Rouet, and Durik (2018) argue that MD reading, like problem-solving (e.g. Funke and Frensch 2007), is a goal-directed activity to reach a specific desired state. However, a reader’s goal in MD situations cannot be achieved with reading routines alone. Instead, MD situations typically involve highly interconnected elements (Hahnel, Schoor et al. 2019) and require an interplay of knowledge acquisition and goal-oriented knowledge application (see Fischer, Greiff, and Funke 2012): Knowledge about an MD situation comes through reading and synthesis, but knowledge also informs throughout the reading activity what actions are needed to progress towards the desired goal (e.g. corroborating information with other documents, evaluating information in light of its source).

Although not every MD situation requires the same actions, in the university context, MD reading benefits from both highly automated routines and controlled processes of reading. Reading is generally considered a complex construction process that relies on several cognitive processes (Perfetti and Stafura 2013). For experienced readers, many of these processes are highly automated (e.g. lower-level processes of word identification), whereas others (e.g. controlled higher-order processes of inferencing and knowledge integration) are slower and more attention-demanding (see Samuels and Flor 1997). The present study focuses on individual differences in serval aspects of university students’ MD reading with respect to different cognitive skills. These are (1) general reading speed, which expresses the level of automation of basic reading skills, and (2) working memory and single-text comprehension as central elements of controlled processing. Specifically, we consider students’ comprehension of MDs and indicators of their strategic processing (i.e. time allocation, corroboration, sourcing). Time allocation is defined as students’ engagement during a task (Naumann 2019); corroboration describes a heuristic of checking multiple sources of information against each other for consistency and plausibility; and sourcing summarizes reader actions of paying attention to meta-information of documents early on and using it to evaluate and contextualize a described claim or situation (Wineburg 1991; also ‘proactive sourcing’ in Hahnel, Kroehne et al. 2019). In summary, the present study investigates the extent to which variables of MD reading are independently predicted by the cognitive skills of reading speed, working memory, and single-text comprehension. Moreover, we examine to what extent indicators of students’ work behavior predict MD comprehension over and above the investigated cognitive skills. Previous research (e.g. Florit, Cain, and Mason 2019; Hahnel, Schoor et al. 2019) shows positive predictions of students’ MD performance by their actions of corroboration and sourcing. However, specific and unique contributions beyond students’ cognitive skills are yet to be explored.

Automaticity in multiple document reading

Accurate reading requires attention, but not for automated reading processes (Samuels and Flor 1997). Automaticity develops over time with extended practice, allowing several tasks (e.g. decoding words, relating information within and across sentences) to be performed simultaneously, without the conscious application of the required subskills. Once lower-level reading processes are automated—or verbally efficient (e.g. Perfetti 2007)—they are assumed to preserve cognitive resources, which readers can devote to higher-level processes that cannot be automatized. Accordingly, inefficiently executed lower-level processes hinder comprehension by requiring attention and energy. As such, reading fluency remains an important predictor of reading comprehension even with advancing reading experience (Florit and Cain 2011; Hannon 2012), with faster readers typically demonstrating higher comprehension skill (for an overview, see Perfetti 2007). In contrast, in problem-solving tasks, individuals need to closely monitor and eventually adjust their comprehension of a situation in a controlled way, therefore allocating more time for processing and increasing the probability for success (Goldhammer et al. 2014). This is reflected, for example, in the positive time-on-task effect and the positive effect of single-text comprehension on time allocation in digital reading tasks, where navigational demands require readers to actively decide on their information processing (Naumann and Goldhammer 2017; Naumann 2019).

Concerning automated reading processes and MD reading, previous research shows that MD comprehension is associated with word recognition and word reading fluency in school students (Bråten et al. 2013; Florit, Cain, and Mason 2019). This relationship was independent of how students had read MDs (i.e. in a linear or non-linear fashion; Bråten et al. 2013). However, the relationship between MD reading and word recognition does not seem to be based on information integration within and between documents, or on the assessment of a document’s trustworthiness, with empirical work finding no association between these components (Strømsø, Bråten, and Samuelstuen 2008; Braasch et al. 2014). Yet, hypertext research indicates that basic reading skills assist readers in locating information and assessing its relevance to a task at hand (e.g. Rouet et al. 2011). Hence, lower-level skills might support readers in identifying potentially related passages across documents (e.g. recognizing ‘predictor’, ‘covariate’ or ‘feature’ as synonyms for ‘independent variable’ when learning about regression analysis) rather than connecting ideas (Schoor, Melzner, and Artelt 2019). In this respect, automated basic reading skills can be expected to support readers when corroborating information across documents, but not in making use of source information that predominantly requires knowledge integration.

Controlled processing

Beyond automated reading components, it is reasonable to assume that most MD processes require attention, effort, and controlled processing. Since MDs often lack cohesive means (e.g. ‘text 1 is related to text 2 in that … ’) to create coherence between documents (e.g. does a situation from one text correspond with what is described in another text?), readers need to be capable of adequately assessing, monitoring, and adjusting their comprehension if necessary. Working memory is a critical resource for these processes. It provides the mental capacity for performing controlled processes, such as reading comprehension (Just and Carpenter 1992) or problem-solving operations (Wiley and Jarosz 2012). The working memory concept refers to a hypothetical system needed to retrieve information from long-term memory and to temporarily store and actively manipulate it for executing complex cognition (Wilhelm, Hildebrandt, and Oberauer 2013). Accordingly, it is used to rapidly create, maintain, and update arbitrary bindings in order to construct and manipulate representations of processed information or content (e.g. words are bound to positions in a syntactical and propositional schema).

Research that relates MD reading to working memory is still scarce. One study on MD reading in a hypertext environment suggests that readers of higher working memory capacity might be in a better position to make global, not explicitly stated connections (see Barzilai and Strømsø 2018). However, Hahnel et al. (2017) argued that in hypertext environments working memory demands, beyond those of text comprehension (e.g. Hannon 2012), arise from the necessity of navigational decision-making. Accordingly, in MD reading situations without complex navigation, an involvement of working memory resources beyond controlled processes of text comprehension might not be expected. This view is consistent with the findings of Florit, Cain, and Mason (2019). They showed that single-text comprehension and the use of source information are important predictors of MD comprehension in primary school children, whereas verbal working memory did not contribute uniquely.

Present study

Contributing to our understanding of cognitive processes in MD reading for higher education learning, the present study investigated how MD reading relates to cognitive skills that are central for single-text reading and require attention to varying degrees. We examined how cognitive skills, which reflect students’ potential of automated (reading speed) and controlled processing (working memory, single-text reading comprehension), are associated with characteristics of their MD work process (time allocation, corroboration, sourcing) and MD comprehension. Specifically, we examined the following hypotheses, considering general cognitive performance (indicated by school graduation grades) and familiarity with study-related MD tasks (indicated by study level; see Hahnel, Schoor et al. 2019) as control variables.

H1. Time allocation in an MD task is negatively predicted by reading speed, but positively by reading comprehension.

H2. Corroboration is positively predicted by reading speed and reading comprehension.

H3. Proactive sourcing is positively predicted by reading comprehension of single texts only.

H4a. Reading speed and working memory positively predict MD comprehension.

H4b. The effect of working memory, but not reading speed, is completely mediated by reading comprehension.

H4c. The characteristics of students’ work process explain MD comprehension over and above the cognitive skills.

Method

Sample and procedure

We recruited a convenience sample of 508 students enrolled in programs of the social sciences and humanities at four German universities (78.0% female; M_age = 22.8, SD_age = 3.8; Bachelor: 53.3%; M_grade = 2.1¹, SD_grade = 0.7). Due to technical issues (e.g. server connection problems), some data parts were not recorded or reliable, resulting in 401 complete cases (78.6% female; M_age = 22.4, SD_age = 3.7; Bachelor: 57.1%; M_grade = 2.1, SD_grade = 0.7). Students completed a questionnaire capturing demographic information (e.g. study level, graduation grades) and three major test parts to assess their MD comprehension, single-text reading skills (reading comprehension, reading speed), and working memory. Participants worked independently on the computer under the supervision of trained test administrators. For the MD assessment part, students worked on two units (i.e. sets of two or three documents including comprehension questions) randomly assigned from a pool of five units according to a balanced incomplete block design. The order of the three major test parts was counterbalanced across participants and students were free to take a break between parts and between the assigned MD units. The session ended after the last major part (duration approx. 2 h).

Measures

An overview of all measures is provided in Table 1.

Multiple document comprehension

MD comprehension was assessed with the MDC test of Schoor et al. (2020; also Schoor et al. 2020; Figure 1). The test comprised five units with 67 comprehension items. The documents covered topics with mostly fictitious contents (e.g. descriptions of an event in the year 2134) and mostly complemented each other (i.e. contradictory information was sparse, but present at detail level). The MDC units allowed free navigation between documents and items, accessing source information about documents, and adding highlights and comments. Two units included an additional essay writing task. At the beginning of each unit, students were provided with an overall reading goal, the number of documents and items, and information about unit-specific time limits (between 27–38 min). All functionalities were introduced in a video-supported tutorial.

Figure 1. Example unit of the MDC test.

MDC item scores

The MDC items required students to (1) corroborate information across documents, (2) integrate information across documents, (3) compare and evaluate sources of documents, or (4) compare and evaluate document contents in light of their sources. Response formats included single-choice formats (1 of 4 and true/false). Dichotomously scored item responses (0–incorrect, 1–correct) served as a dependent variable in the present study (correct response rates: 17.0% to 97.5%).

Student behavior

Students’ interactions during the test were recorded in log files, allowing the derivation of process indicators. The R package LogFSM (Kroehne 2020) was used to determine students’ unit processing time in minutes (time allocation), the number of switches between documents per unit (corroboration), and a dichotomous variable indicating whether at least two sources in a unit were visited before the documents were fully processed (proactive sourcing, see Hahnel, Kroehne et al. 2019).

Reading speed

A sentence verification task was used to measure basic reading processes of lexical access and proposition integration at sentence level (Zimmermann, Artelt, and Weinert 2014). The task requested students to evaluate 51 sentences (e.g. ‘There is a bath tub in every garage.’) as ‘true’ or ‘false’ as accurately and quickly as possible. The sum score of correctly evaluated sentences within a 2-minute time limit served as indicator of students’ reading speed.

Working memory

A verbal and a figural updating task were used to capture students’ skill of storing and manipulating information in their working memory (Wilhelm, Hildebrandt, and Oberauer 2013). In the verbal updating task (12 trials), students had to memorize two to five words associated with a set of categories (e.g. bird and lemon for the categories animal and fruit). After a short presentation time (2000ms), up to four words per category were presented successively (e.g. horse–mango–apple–cat). The participants were asked to recall the last word of each category (i.e. cat for animal, apple for fruit). The figural updating task (11 trials) worked similarly, but presented two to five colored squares in a 3 × 3-grid (e.g. red square top right, blue square middle bottom) and the participants were asked for the last position of each color.

Based on the percentage of words or positions correctly recalled in each trial, we estimated a confirmatory factor model (n = 487 cases) with the R package lavaan (Rosseel 2012) using a full information maximum likelihood approach (FIML). All items loaded on one common latent factor and on one of two uncorrelated bi-factors (i.e. verbal vs. figural task material). The model fit was good, χ²(249) = 293.87, p = .027, RMSEA = .02, CFI = .97, SMSR = .03. The factor scores of the common latent variable severed as indicator of working memory.

Reading comprehension

Students’ skill to comprehend written text was assessed with the reading comprehension test of the National Educational Panel Study (NEPS; Gehrer et al. 2013). Similar to the MDC test, this test was organized in five units including one text and several comprehension items. In contrast, the students were asked to complete all units within an overall 28-minute time limit. The test functionalities were explained in a tutorial.

A total of 21 items required students to (1) find information within the text, (2) interpret and draw inferences, and (3) reflect on and evaluate the information of the text. The item response formats included single-choice (e.g. 1 of 4) and matching formats (e.g. matching headings with paragraphs). Students’ item responses were scored dichotomously, or in parts with a partial credit solution (e.g. 0–incorrect, 1–partial credit, 2–correct). We fitted a Partial Credit Model (n = 506 cases) with the R package TAM (Robitzsch, Kiefer, and Wu 2019) and obtained Weighted Likelihood Estimates (WLEs) as indicator of students’ reading comprehension (WLE reliability = .71). A previous study demonstrated that the NEPS reading comprehension and the MDC test assess highly correlated, but independent skills (Mahlow et al. 2020).

Data analysis

Linear mixed models (LMMs) and generalized linear mixed models (GLMMs) were used to predict students’ MD reading behavior (time allocation, corroboration, sourcing) and their probability of success in the MDC items with the R package lme4 (Bates et al. 2015). All models included fixed effects for the MDC units, the unit position in the MDC test (unit position), position in the overall procedure (test position), graduation grades and study level (reference category: bachelor). To consider the hierarchical data structure, random intercepts were modeled for students and, in case of predicting MD task success, items. All continuous predictors were z-standardized. The time allocation and corroboration variables were added with a constant of 1 and log-transformed before standardization. Below, we report standardized regression coefficients (β) for the LMMs and odds ratios (OR) for the GLMMs.

Results

The descriptive statistics for all variables are displayed in the Tables 1 and 2. The predictions of MD work behavior are summarized in Table 3. As expected (H1), time allocation was negatively predicted by reading speed (β = −.09) and positively by reading comprehension (β = .15). Interestingly, students with higher working memory took significantly longer to process the MDC units (β = .13) and master students needed less time than bachelor students (β = −.09). Corroboration was positively related to both reading speed (β = .08) and reading comprehension (β = .17), and the probability of showing proactive sourcing was predicted by reading comprehension only (OR = 3.19), supporting H2 and H3. Notably, master students were more likely to show behaviors of corroboration (β = .10) and proactive sourcing (OR = 8.64) than bachelor students. Finally, significant position effects indicate that students tended to process the units faster over time (β = −.09 to −.37). Proactive sourcing was more likely in the second MDC unit (OR = 20.39), whereas MDC-specific behaviors were less frequent at the end of the entire test procedure (β = −.27 and OR = 0.07).

Table 1. Overview of the study variables.

Variable	Operationalization	Level of aggregation	Level of measurement	Value description	M/RF	SD	Skew	Kurtosis
Indicators of multiple document (MD) reading
MDC task success	item scores in the MDC test	item	dichotomous	0 = incorrect item response 1 = correct item response	0.66	-	-	-
time allocation	unit processing time	unit	continuous	high values indicate longer processing times (minutes)	23.71	8.86	0.25	−0.53
corroboration	number of switches between documents	unit	continuous	high values indicate more frequent document switches	11.76	7.36	1.16	2.37
proactive sourcing	indicator of whether the sources of at least two documents were accessed within the first 10% of the processing time of the respective document	unit	dichotomous	0 = less than two sources in a unit were visited proactively 1 = at least two sources in a unit were visited proactively	0.20	-	-	-
Fundamental cognitive skills
reading speed	sum scores of correctly evaluated sentences within a 2-minute time limit	person	continuous	high values indicate higher speed	45.58	5.81	−1.24	1.00
working memory	factor scores from a confirmatory factor model	person	continuous	high values indicate higher proficiency	0.05	5.24	−2.66	8.79
reading comprehension	weighted likelihood estimates from a Partial Credit Model	person	continuous	high values indicate higher proficiency	−0.02	0.94	0.57	5.60
Control variables
general cognitive performance	German school graduation grades (‘Abiturnote’)	person	continuous	small values indicate higher proficiency	2.13	0.66	0.07	−0.94
familiarity with study-related MD tasks	study level	person	dichotomous	0 = Bachelor program 1 = Master program	0.43	-	-	-
unit	assigned unit of the MDC test	unit	nominal	0 = MDC unit ‘2134’ 1 = MDC unit ‘Catalano’ 2 = MDC unit ‘Nothing’ 3 = MDC unit ‘Animals’ 4 = MDC unit ‘Universe’	-	-	-	-
unit position	position of the assigned unit within the MDC test part	unit	dichotomous	0 = first unit 1 = second unit	0.50	-	-	-
test position	position of the MDC test part within the overall test procedure	unit	ordinal	0 = first position 1 = second position 2 = third position	-	-	-	-

Notes. M and SD are the mean and standard deviation of the unstandardized variables. RF is the relative frequency of category ‘1’ of dichotomous variables.

Table 2. Correlations on person level (n = 401 students).

Variables		(1)	(2)	(3)	(4)	(5)	(6)	(7)
(1)	general cognitive performance (graduation grades)^1	-
(2)	familiarity with study-related MD tasks (study level)	.05	-
(3)	reading speed	−.21***	−.08	-
(4)	working memory	−.13*	.03	.21***	-
(5)	reading comprehension	−.32***	.17***	.29***	.35***	-
(6)	time allocation^2	−.09	−.09	−.04	.15**	.16**	-
(7)	corroboration^2	−.17***	.14**	.16**	.19***	.29***	.55***	-
(8)	proactive sourcing^3	−.14**	.21***	.10*	.05	.25***	.10	.22***

Notes. ¹Small values indicate higher proficiency. ²For determining correlations on person level, these variables were aggregated by averaging log-transformed values over units. ³For determining correlations on person level, this variable was aggregated by the sum of values over units.

*p < .05, **p < .01, ***p < .001.

Table 3. Prediction of time allocation (LMM), corroboration (LMM), and sourcing (GLMM).

	Time allocation (H1)		Corroboration (H2)		Proactive sourcing (H3)
	Est. (SE)	β	Est. (SE)	β	Est. (SE)	OR
intercept	−0.16 (0.08)		0.25 (0.10)		−7.32 (0.51)
unit position 2	−0.19 (0.04)	−.09***	−0.06 (0.05)	−.03	3.01 (0.26)	20.39***
test position 2	−0.25 (0.09)	−.12**	−0.12 (0.09)	−.06	−1.29 (0.38)	0.28
test position 3	−0.79 (0.09)	−.37***	−0.57 (0.10)	−.27***	−2.65 (0.42)	0.07***
graduation grades	−0.03 (0.04)	−.03	−0.08 (0.04)	−.08	−0.48 (0.19)	0.62
study level Master	−0.17 (0.07)	−.09*	0.20 (0.08)	.10*	2.16 (0.36)	8.64***
Cognitive skills
reading speed	−0.10 (0.04)	−.09*	0.08 (0.04)	.08*	0.31 (0.18)	1.36
working memory	0.13 (0.04)	.13**	0.06 (0.04)	.06	−0.27 (0.17)	0.76
reading comprehension	0.15 (0.04)	.15***	0.18 (0.05)	.17***	1.16 (0.21)	3.19***
% variance explained	10.5		19.5		57.9

Note. Unit effects are not reported. Est. = unstandardized coefficients; β = standardized coefficients; OR = Odds ratio.

*p < .05, **p < .01, ***p < .001.

The prediction of MD task success is summarized in Table 4. Supporting H4a, students were more likely to solve a MDC item if they were fast readers (OR = 1.15) and possessed higher working memory skill (OR = 1.11). As expected (H4b), adding reading comprehension (OR = 1.40) canceled the working memory effect, whereas the reading speed effect remained significant (OR = 1.08). Adding the process indicators (H4c) showed that time allocation (OR = 1.12), corroboration (OR = 1.10) and proactive sourcing (OR = 1.16) contributed independently to the prediction of students’ MDC task success. Notably, students performed better in the second MDC unit (OR = 1.19), and having better grades (OR = 0.89) or being a master student (OR = 1.17) was also associated with higher MD comprehension.

Table 4. Prediction of the probability of MDC task success (GLMMs).

	H4a		H4b		H4c
	Est. (SE)	OR	Est. (SE)	OR	Est. (SE)	OR
intercept	0.88 (0.29)		0.93 (0.29)		0.88 (0.29)
unit position 2	0.16 (0.05)	1.17***	0.16 (0.05)	1.17***	0.17 (0.05)	1.19***
test position 2	−0.09 (0.08)	0.91	−0.04 (0.07)	0.96	0.00 (0.07)	1.00
test position 3	−0.25 (0.08)	0.78**	−0.27 (0.08)	0.76***	−0.10 (0.08)	0.90
graduation grades	−0.20 (0.03)	0.82***	−0.12 (0.03)	0.88***	−0.11 (0.03)	0.89***
study level Master	0.29 (0.07)	1.34***	0.18 (0.06)	1.19**	0.16 (0.06)	1.17*
Cognitive skills
reading speed	0.14 (0.04)	1.15***	0.07 (0.03)	1.08*	0.07 (0.03)	1.08*
working memory	0.10 (0.03)	1.11**	0.02 (0.03)	1.02	0.00 (0.03)	1.00
reading comprehension	-		0.33 (0.04)	1.40***	0.29 (0.04)	1.33***
Indicators of behavior
time allocation	-		-		0.12 (0.04)	1.12**
corroboration	-		-		0.09 (0.03)	1.10**
proactive sourcing	-		-		0.15 (0.07)	1.16*
% variance explained	33.7		54.2		59.6

Note. Unit effects are not reported. Est. = unstandardized coefficients; OR = Odds ratio.

*p < .05, **p < .01, ***p < .001.

Discussion

We investigated automated and controlled processes in MD reading by examining individual differences in university students’ work behavior and their comprehension of MDs with respect to reading speed, working memory, and reading comprehension of single texts. The results of our analyses support the hypotheses outlined above. In the following, we discuss the evidence our findings provide about automated and controlled processes in MD reading. We also discuss ancillary findings that were not the primary focus of this study but may contribute insights to closely related issues of MD reading. Finally, we conclude the section with a reflection on the study’s limitations and general conclusions.

Automated and controlled processing in multiple document reading

On the background of automaticity in reading (Samuels and Flor 1997; Perfetti 2007), the study’s results show strong evidence for the impact of automated processes in MD reading, as indicated by the results involving reading speed. Faster readers, whom we assumed to possess highly automated basic reading skills, processed MD tasks faster, engaged in more corroboration behavior (but not sourcing behavior), and showed better comprehension of MDs. These effects even persisted when reading comprehension and behavior during task processing were considered. In this regard, our results are in line with previous research showing that word recognition supports MD comprehension (e.g. Bråten et al. 2013). They also bridge the gap to other studies (e.g. Strømsø, Bråten, and Samuelstuen 2008; Braasch et al. 2014) by suggesting that basic reading skills indirectly assist information integration by supporting readers in locating related information across documents (see Rouet et al. 2011).

In comparison to investigating automated processes, mechanisms of controlled processing became evident in the effect patterns involving working memory and reading comprehension. Concerning MD reading behaviors, readers with higher single-text comprehension skills, whom we assumed better at identifying needs for controlled processing, such as monitoring, inferencing, and information integration (Britt, Rouet, and Durik 2018; Naumann 2019), showed the expected differences. They spend on average longer on MD tasks and engaged in more corroboration and sourcing behavior than less skilled readers, suggesting that they were better in recognizing specific requirements of MD reading and reacted to them strategically. However, the positive working memory effect on processing time was surprising. It shows that time allocation of two students with comparable cognitive skills still differs with respect to their working memory resources. It is rather unlikely that this effect reflects disengaged test-taking, training, or exhaustion, since we considered position effects when modeling this relationship. Yet, it might indicate differences in effort, in the sense that, when reading about an unfamiliar topic, students with lower working memory resources might set lower standards for their MD comprehension in order to cope with the required cognitive demands (see Hahnel, Schoor et al. 2019). However, this is a speculation that needs thorough investigation.

In terms of performance, better single-text comprehension skills predicted better comprehension of MDs while fully meditating the effect of working memory on MD comprehension. This result corresponds with findings on primary school children (Florit, Cain, and Mason 2019) and supports the view that, without a complex navigational structure, working memory demands in MD reading do not exceed those of text comprehension (Hahnel et al. 2017; cf. Barzilai and Strømsø 2018; Mahlow et al. 2020). Moreover, the indicators of students’ work process uniquely contributed to the prediction of MD comprehension beyond cognitive skills. In line with work showing their predictive value (e.g. Hahnel, Schoor et al. 2019), this result supports the view that MD reading requires MD-specific strategic processing (see Cho, Afflerbach, and Han 2018; Schoor et al. 2021).

Furthermore, while not an explicit part of our investigation, one might have expected that proximal behavioral indicators mediate the effect of distal cognitive skills on performance (see Hahnel et al. 2017)—a reasonable view from a general perspective on item processing. However, current theories of MD reading do not provide indication for a full mediation effect involving reading-related skills as predictors and MD-specific behaviors as mediators (for an overview of theoretical perspectives on MDC, see Mahlow et al. 2020). Our results also indicate against such a mediating effect. Although we did not examine indirect effects, the comparison of the Models H4b and H4c demonstrates that the effects of the cognitive skills remained relatively unchanged after adding the behavioral variables. A substantial decrease in the regression coefficients should have been observed in the case of a mediation effect.

Notable ancillary findings

Although not the primary focus of our study, our results provide interesting ancillary findings regarding (1) the effects of students’ graduation grades and study levels and (2) the positive effects of unit position on MDC task success. Concerning the first, German graduation grades are an aggregation of grades assessed by several teachers over two years. With this in mind, it might not be surprising that graduation grades uniquely contributed to MD comprehension, although several cognitive skills were considered. However, they were not predictive for student behavior. This emphasizes the notion that a comprehensive understanding of MDs can be achieved by behaviors that can be learned and strategically applied (Britt and Rouet 2012; Hahnel, Schoor et al. 2019), which is also reflected in the finding that master students spent less time with MD task processing, but showed more favorable MD-specific behavior than bachelor students. Acknowledging these differences, though, master students still outperformed bachelor students in MD comprehension, suggesting that the experiences they made throughout their studies might have included other mastery and proficiency gains that supported their comprehension of MDs. However, graduation grades and study level are relatively distal indicators of general cognitive performance and familiarity with study-related MD tasks. Accordingly, the relationship of MD reading with other operationalizations (e.g. reasoning skill or actual student experience with MD tasks) should be explored in future research.

Concerning unit position, we observed that it was easier for students to solve items of the second MD unit than those of the first. This position effect might merely indicate that students have adopted practices that improved their performance, without actual increases in the skill measured (test wisdom; see Downing and Haladyna 2006) or that have been acquired in similar prior situations (context schema formation; see Britt, Rouet, and Durik 2018; Schoor et al. 2021). However, viewing MD reading as a problem-solving process in light of Ackerman’s three-phase theory (see Ackerman 2004), it could indicate actual learning, with MD reading requiring lower levels of attention and effort from students as they transition into an associative stage of skill acquisition. However, to investigate such an effect in detail, a thoughtful pre–post or longitudinal design with more than two measurement time points would be appropriate. Furthermore, we based our study on the assumption that MD reading resembles a problem-solving process without addressing this assumption empirically. We see worthwhile insights in this direction, but the theoretical link between both needs to be empirically substantiated.

Limitations

Apart from the already mentioned restrictions, there are further limitations to our study. One of these is that the behavioral indicators examined are not an ideal representation of conscious strategic activities (see Cho, Afflerbach, and Han 2018). Similarly, this also applies to the reading speed measure since speed is not the same as efficiency or fluency (Perfetti 2007). Accordingly, our results need replication involving other measures and indicators of both automated and controlled processing. On another note, we investigated a convenience sample of German students from the social sciences and humanities. Although we were able to examine a fairly large sample despite technical issues, the generalizability of our results to other student groups remains limited.

Conclusion

Our findings highlight that MD comprehension depends on a number of cognitive skills and behaviors that are also affected by those skills. In light of this Special Issue on Progressions in Learning in the Age of (Mis)Information, our findings point to an urgent need for students to have profound skills in working with multiple documents or multiple sources, as existing problems may intensify if learning materials increasingly include online information without formal approval. Some students already face problems with developing a deep understanding of a topic using MDs, even without encountering requirements such as selecting information sources, assessing their trustworthiness, or resolving contradictions between conflicting claims. Accordingly, comprehending MDs is not a skill we can expect university students to have mastered already, making it necessary to provide them with systematic guidance and learning opportunities for further development.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by the German Federal Ministry of Education and Research under [Grant no 01PK15008].

Notes

1 School graduation grades—the German ‘Abitur’—range from 1–‘very good’ to 4–‘sufficient’.