Contributions of Academic Language, Perspective Taking, and Complex Reasoning to Deep Reading Comprehension

ABSTRACT

Deep reading comprehension refers to the process required to succeed at tasks defined by the Common Core State Literacy Standards, as well as to achieve proficiency on the more challenging reading tasks in the Program for International Student Assessment (PISA) framework. The purpose of this study was to test the hypothesis that three skill domains not frequently attended to in instruction or in theories of reading comprehension—academic language, perspective taking, and complex reasoning—predict outcomes on an assessment of deep reading comprehension. The Global Integrated Scenario-based Assessment (GISA; O'Reilly, Weeks, Sabatini, Halderman, & Steinberg, 2014) is designed to reflect students' abilities to evaluate texts, integrate information from an array of texts, and use textual evidence to formulate a position, all features of deep reading comprehension. We tested the role of academic language, perspective taking, and complex reasoning in explaining variance in end-of-year GISA scores, controlling for beginning-of-year scores and student demographics. All three predictors explained small, but significant, amounts of additional variance. We suggest that these three skill domains deserve greater attention in theories of reading comprehension and in instruction.

KEYWORDS:

• reading comprehension
• academic language
• perspective taking

Reading comprehension is undeniably the literacy challenge of the 21st century. A half-century of systematic research has provided strong guidance to instruction in word reading accuracy and fluency (National Institute of Child Health and Human Development [NICHD], 2000). Whereas U.S. students perform well in international comparisons in fourth grade (Martin & Mullis, 2013) when these technical reading skills are the prime determinant of successful performance, they perform relatively poorly by high school when comprehension tasks are more challenging (National Center for Education Statistics [NCES], 2015). Unfortunately, research has generated much less consensus about either the cognitive processes involved in or the best instructional approaches to successful reading comprehension (RAND Reading Study Group, 2002).

In a notable exception to this generalization, the simple view of reading (SVR) has achieved widespread acceptance. The SVR (Gough & Tunmer, 1986) is, simply, that comprehension is the product of decoding and oral language comprehension. In other words, any text that a student can understand orally also can be understood through reading, if decoding of words in that text does not form a barrier. This view had the important effect of directing attention to oral language as a crucial predictor of comprehension outcomes. It has generated dozens of studies, most of which have generally confirmed the predictions of the simple view, across several populations, including: monolingual normally developing participants reading in English (e.g., Savage, 2001) and other languages (e.g., Gentaz, Sprenger-Charolles, Theurel, & Colé, 2013; Ho, Chow, Wong, Waye, & Bishop, 2012), second language readers of English (e.g., Erdos, Genesee, Savage, & Haigh, 2010, Farnia & Geva, 2013; Proctor, August, Carlo, & Snow, 2006), and children with a variety of language and cognitive disabilities (e.g., Catts, Adlof, & Weismer, 2006; Palikara, Dockrell, & Lindsay, 2011). These various studies have operationalized the key variables—decoding, oral comprehension, and reading comprehension—in a variety of ways, and most have been conducted with students reading at or below fourth grade level, a level at which comprehension is assessed using fairly simple texts and relatively low-inference comprehension items. In fact, decoding skill has been shown to explain a large percent of the variance on many of the widely used comprehension assessments, especially for younger readers (Cutting & Scarborough, 2006).

Despite its success, the SVR has been criticized for its failure to recognize the role of broader cognitive abilities that are strongly related to comprehension, including memory and word knowledge (Carrroll, 1993), IQ (e.g., Tiu, Thompson, & Lewis, 2003), efficiency (Høien-Tengesdal & Høien, 2012), and fluency (Silverman, Speece, Harring, & Ritchey, 2013), particularly in accounting for older students' reading (Macaruso & Shankweiler, 2010). Furthermore, the SVR encounters limitations when extended to explaining performance on the literacy tasks of adolescence, the comprehension outcomes defined as expected of all students by the Common Core State Standards (CCSS; National Governors Association [NGA], 2010), and the content-area-specific challenges to comprehension described by those studying disciplinary literacy (Goldman, 2012; Goldman & Snow, 2015; Lee, 2004; Shanahan, Shanahan & Misischia, 2011). It is these limitations that led us to propose an enriched model of reading comprehension, a model to explain what we call deep reading comprehension. The comprehension tasks encountered in secondary school, higher education, and many employment settings require acquisition of knowledge, integration of newly acquired with preexisting conceptual structures, analysis and critique of texts and their sources, and synthesis across multiple texts and sources. We hypothesize that success at these deep reading tasks are, in turn, dependent on abilities in three domains that go well beyond decoding and oral comprehension: academic language, perspective taking, and complex reasoning.

The purpose of the study presented here is to report the first findings testing this hypothesis about the predictors of deep reading comprehension. The findings derived from a study conducted under the Reading for Understanding initiative, in the context of a larger effort to evaluate instructional interventions to promote deep comprehension skills. In this report, though, we focus on examining relations among predictors and outcomes among students who did not receive the intervention, rather than testing intervention effects. We start by exploring the characteristics of deep reading comprehension and ways of assessing it, and then go on to review prior research supporting the relationship of academic language, perspective taking, and complex reasoning to comprehension.

Deep Reading Comprehension

Literacy tasks envisioned in the CCSS (NGA, 2010) include reading to collect textual evidence in support of a claim, inferring word meanings from close reading of text, integrating information about a topic from multiple texts, and reading to critique others' arguments (e.g., sixth- through eighth-grade literacy in history/social studies reading standards 1, 4, and 8, respectively). Such tasks are considered good preparation for the college- and career-ready literacy challenges toward which the Common Core Standards build. These challenges also are reflected in the expectations of the 2012 Program of International Student Assessment (PISA) Reading Framework, which defines the highest level of proficiency as follows:

Tasks at this level typically require the reader to make multiple inferences, comparisons, and contrasts that are both detailed and precise. They require demonstration of a full and detailed understanding of one or more texts and may involve integrating information from more than one text. Tasks may require the reader to deal with unfamiliar ideas, in the presence of prominent competing information, and to generate abstract categories for interpretations. Reflect and evaluate tasks may require the reader to hypothesize about or critically evaluate a complex text on an unfamiliar topic, taking into account multiple criteria or perspectives, and applying sophisticated understandings from beyond the text. A salient condition for access and retrieve tasks at this level is precision of analysis and fine attention to detail that is inconspicuous in the texts. (Organisation for Economic Co-operation and Development [OECD], 2013, p. 79)

The difficulty of preparing students to be highly proficient at literacy tasks such as those required by PISA and the CCSS is clear from the finding that only 0.8% of U.S. high school students tested by PISA perform at this level. So the questions that arise are what instructional approaches might help students meet these challenges, and what skills those 0.8% possess that less skilled students lack.

The default in comprehension instruction in the United States is teaching comprehension strategies, the approach endorsed by the National Reading Panel (NICHD, 2000). At least before the advent of professional development and curricula aligned with the CCSS, there was little insistence in most U.S. schools that students grapple with critically evaluating complex texts on unfamiliar topics, generating abstract categories for interpretation, or sifting through competing information. Perhaps the primary locus for actual training in tasks such as these in U.S. K–12 education has been Advanced Placement history and literature courses (Young & Leinhardt, 1998), courses to which only a small minority of students have access.

One reason that deep reading comprehension has received so little instructional attention may be that standardized assessments of reading comprehension rarely measure deep comprehension skills. Emphasizing deep comprehension may require features that have traditionally been difficult to incorporate into relatively brief, group-administered, machine-scorable assessments, such as the use of multiple texts and the option of open-ended responses with unconstrained themes or formats (Sabatini, Petscher, O'Reilly, & Truckenmiller, 2015). Furthermore, interpreting performance on deep reading comprehension assessments may require ancillary test data, for example, an evaluation of respondents' background knowledge about the topic.

Fortunately for the program of research reported on here, novel and much more informative assessments have been made possible by the expansion of digital capabilities. For example, Educational Testing Service (ETS) has developed an innovative suite of computer-delivered assessments called the Global Integrated Scenario-based Assessments (GISA). GISA is a theoretically based measure designed to reflect an updated understanding of the construct of reading comprehension while taking advantage of the affordances of technological delivery (O'Reilly & Sabatini, 2013; Sabatini & O'Reilly, 2013; Sabatini, O'Reilly, & Deane, 2013). In the study reported here, we use the GISA as our outcome measure of deep reading comprehension (see Methods section for a fuller description of the test and its psychometric properties).

Predictors of Deep Reading Comprehension

Hypotheses about the specific skills that need to be added to the SVR to explain deep reading comprehension derive from a variety of sources: an analysis of deep reading tasks, evidence about what professional historians, scientists, and literary analysts do when reading, evidence about factors that increase processing load while reading, and hints derived from the content of effective programs of professional development and instruction for (post)adolescent readers. In the brief review that follows, we suggest how these various sources contributed to generating the hypothesis to be tested in this study.

Academic Language

Academic language is now widely cited as a literacy challenge for many students. Precise definitions or conceptualizations of academic language have varied widely (Snow & Uccelli, 2009), and in the practice and professional development literature academic language is still often primarily identified with use of “academic vocabulary,” itself a construct based on corpus analysis (Coxhead, 2000) rather than theory. There is a long history of research within sociolinguistics and discourse analysis documenting the ways in which written language differs from oral and formal registers differ from informal, all of which are relevant to defining the endpoints of the continuum from casual to academic language. Key features of academic language, described by Halliday (1987, 1993), Schleppegrell (2001, 2007), Scarcella (2003), and others, include reduction in use of subject pronouns and action verbs; increase in nominalizations, passives, and embedded relative clauses; and lexicalized discourse, stance, and epistemic markers. These features have been shown to increase processing burden during reading (Carpenter & Just, 1989), in particular for readers who have not encountered them frequently.

Schleppegrell has built programs of professional development around the analysis of academic language features in high school content-area texts to display to teachers constructions and linguistic usages that might well be unfamiliar and puzzling to their students (e.g., Schleppegrell, Achugar, & Oteiza, 2004; Schleppegrell & de Oliveira, 2006). The close reading practices encouraged by the CCSS teaching guidelines also may be useful in focusing student attention on the academic language structures in their texts. Instruction in writing often provides students with model texts and sentence starters that display how academic language is used. To our knowledge, though, in none of these instructional approaches have improvements in academic language skills been specifically related to changes in reading comprehension outcomes.

Perspective Taking

Particularly when reading literary narratives and history, a great challenge is to recognize that different actors have different experiences of the same events. The ability to understand and to navigate those varying perspectives, which we refer to as perspective taking, is critical to understanding literary and historical conflicts, to attributing psychological causality, and to evaluating actions. Chall's (1983) stage theory of reading development identified stage five, which she associated with the high school years, as the stage of multiple perspectives, and a long history of theorizing in literature studies promotes novel reading as a way to enhance perspective taking (Leverage, Mancing, Schweickert, & William, 2011). The relevant empirical literature, on the other hand, is scattered, in part because the construct of perspective taking has been variously operationalized as theory of mind (Pelletier, 2006), narrative empathy (Keen, 2007), social cognition (Shantz, 1982), or social information processing (Donahue, 2001).

Kessler and Donahue (2008) found that performance on a measure of students' social information processing, in particular their ability to encode social cues, predicted processing of character perspectives relevant to understanding a short story's surprise ending. Fewer than half of seventh and eighth grade readers fully understood the surprise, a finding Kessler and Donahue tentatively attribute to the difficulty of resolving the perspectives of three characters with different representations of relevant information. Donahue (2014) hypothesizes that failure of social information processing skills may explain the particular comprehension difficulties of students with learning disabilities, and the unexpectedly good comprehension of second language learners still struggling with reading proficiency (Pelletier, 2006). Gardner and Smith (1987) defined the development of perspective taking within a Piagetian perspective as an increase in the capacity to acknowledge reciprocal relationships. Whereas they found that perspective taking skills have no correlation with reading comprehension as measured by the California Achievement Test, they did find perspective-taking skills to be significantly correlated with deep reading comprehension as measured by responses to script-implicit (i.e., deep inferential) questions about a narrative.

In processing narratives about what happens when one character deceives or inadvertently misleads another, good readers have to keep track of what both characters know. Younger readers/listeners often attribute what they themselves know about the location of objects or contents of a container to characters who have not had access to that information in the narrative (Biancarosa, 2006; Lynch & van den Broek, 2007). Biancarosa demonstrated that better comprehenders slow down when reading sentences that imply characters' access to concealed information, suggesting strongly that good readers keep better track of characters' perspectives than poor readers. These rather simple challenges to perspective taking for younger readers presage the difficulty older readers have in understanding that there are multiple defensible positions on many questions, that people hold those opinions for reasons that have to do with their own life experiences, and that understanding another's point of view neither requires nor excludes agreeing with it.

Complex Reasoning

The third skill hypothesized to relate to deep reading comprehension has to do with the complexity of students' epistemic thought— their reasoning about inquiry, evidence, truth, knowledge, reasoning, conflict, and deliberation. It has been characterized as the ability to think effectively about complex issues that have no single correct answer, such as those involving relationships among food sources, breeding seasons, climatic conditions, and risks of predation that determine growth or decline of a population within a particular biome. Problems such as these involve multiple statistical and transactional connections among events and consequences. Reading an explanation of the carbon cycle, or the decline of the Roman Empire, or the long period of stagflation recently experienced by Japan requires understanding and evaluating claims made with varying degrees of certainty about multiple, interlocking influences. Evidence that tracking explanations of complex phenomena while reading is a severe challenge to students is abundant in teacher reports, and in the work-arounds content area teachers introduce—lectures, PowerPoint presentations—to convey basic information to students unable to process such information by reading their textbooks.

Early research on children's and adolescents' reasoning focused on the changes, described by Piaget, that occurred with the shift to formal operational thinking, theorized to begin at about age 11 or 12 and marked by the ability to use hypothetical and deductive reasoning, among other abstract skills (Inhelder & Piaget, 1958). Later work undertaken by Carey (1985), Vosniadou (2013) and diSessa (1988) has focused on the nature of the conceptual change that has to occur—that is not so much on the reasoning process itself as the knowledge structures that students have to reason with. Much of the relevant work uses a confusing mix of overlapping terms and concepts, such as complex reasoning, scientific reasoning, argumentation, critical thinking, and scientific thinking, among others. Most of the research has focused on how these capacities develop, with surprisingly few studies examining how reasoning skills relate to other developmental outcomes. Nonetheless, it is clear that complex reasoning develops over the course of the life span in a sequence of increasingly complex levels (Perry, 1970), and relates moderately to academic and verbal ability (Wood, 1997).

However, research has shown that reasoning skills are often deficient or lacking for both children and adults (e.g., Kuhn, 1991) and that classroom practices that promote argumentation are rare (Newton, Driver, & Osborne, 1999). Studies in science education found that students struggle with several aspects of reasoning and argumentation, such as finding adequate evidence to support a claim, integrating contradictory evidence, and challenging the claims made by others (Bell, 2004; Cavagnetto, Hand, & Norton-Meier, 2010; Evagorou, Jimenez-Aleixandre, & Osborne, 2012; Jimenez-Aleixandre & Pereiro-Munoz, 2002; Sandoval & Millwood, 2005). However, children's scientific reasoning can be effectively promoted by specific classroom practices, such as inquiry-based instruction (e.g., Gerber, Cavallo, & Marek, 2001) and implementation of classroom strategies for teaching argumentation (Osborne, Erduran, & Simon, 2004). Although it seems obvious that students need reasoning skills to navigate texts reporting complex relationships or phenomena (for example, in history, science, or literature), such as those frequently encountered in the middle grades and higher, there has been little research examining the relationship between reasoning and reading comprehension. In the context of science literacy, however, there has been research demonstrating that students have insufficient skills to evaluate and infer meaning from media reports about science (Norris & Phillips, 1994; Zimmerman, Bisanz, Bisanz, Klein, & Klein, 2001).

Purpose of the Study

This study is designed to test the hypothesis that academic language, perspective taking, and complex reasoning predict outcomes on an assessment of deep reading comprehension. We collected data on academic language, perspective taking, and complex reasoning skills as well as on deep reading comprehension performance from students in grades four through seven participating in a large-scale experimental evaluation of a novel curriculum. Utilizing data from the control group, we start with analyses relating academic language and perspective taking to deep reading comprehension, controlling for demographic variables and grade. We then present a more exploratory model that includes the complex reasoning measure, which for technical reasons is only available on a reduced portion of the sample.

Methods

This study utilizes data from an IES-funded evaluation of Word Generation (WG), a tier one, a discussion-based program for middle school students designed to build academic literacy and academic practices through language arts, math, science, and social studies classes. The WG evaluation is a school-level experimental study that includes 24 schools randomized to treatment and control conditions. The first cohort began in 2011 and participated for three years, and the second cohort began in 2012, participating for two years. Data for the present study are from control schools only (n = 12), in the second year of the study, after both cohorts were enrolled.

Participants

The participants in this study included 2,933 students in 124 classrooms (grades four through seven), from diverse backgrounds reflecting the demographics of the urban and semi-urban communities of the schools (51% female, 40% Black, 28% White, 3% Asian, 27% Latino, 1% Native American/Pacific Islander, 1% mixed race/other, 83% eligible for free/reduced-price lunch, 8% English language learners (ELLs), and 14% with special education classification (Table 1). The study had “exempt status” and no consent process was required because assessments were administered as part of standard educational practices to assess the effectiveness of school curricula. Assessments were administered to all students present on testing days.

Table 1. Sample characteristics (N = 1965).

Variable	%
Female	51%
Black	40%
White	28%
Asian	3%
Latino	27%
Native American/Pacific Islander	1%
Mixed/Other	1%
Free/Reduced-Price Lunch	83%
English Language Learners	8%
Special Education	14%
Grade 4	22%
Grade 5	23%
Grade 6	29%
Grade 7	26%

Measures

Core Academic Language Skills

The Core Academic Language Skills-Instrument (CALS-I) is a group-administered instrument designed to assess core academic language skills (CALS) in grades four through eight. CALS refer to the constellation of high-utility language skills that correspond to linguistic features prevalent in oral and written academic discourse across school content areas, but that are infrequent in colloquial conversations (e.g., logical connectives, such as nevertheless, consequently; structures that pack information densely, such as nominalizations or embedded clauses; markers of organization in argumentative texts, such as first, on the other hand) (Uccelli, Phillips Galloway, Barr, Meneses, & Dobbs, 2015). Each CALS-I form consists of a 50-minute paper-and-pencil test that includes eight tasks: Connecting Ideas, Tracking Themes, Organizing Texts, Breaking Words, Comprehending Sentences, Identifying Definitions, Interpreting Epistemic Stance Markers, and Understanding Metalinguistic Vocabulary. Tasks assess students' skills through a range of multiple-choice, matching, or short written responses.

The development of the CALS-I was based on an extensive literature review, followed by an iterative design process that unfolded in the following sequence: a task design phase and pre-pilot study, a series of qualitative and quantitative pilot studies, an expert review panel, and a norming phase (for more information, see Uccelli, Barr, et al., 2015; Uccelli, Phillips Galloway, et al., 2015). Two forms of the CALS-I were used in this study: CALS-I Form 1 designed for grades four through six (α = .90, number of items = 49) and CALS-I Form 2 for grades seven and eight (α = .86, number of items = 46). The items that were not scored dichotomously as correct/incorrect were rescaled to be between 0 and 1 so all items were equally weighted in estimating the total score. Using Rasch item response theory analysis, factor scores were generated for the CALS-I using a vertically equated scale.

Social Perspective Taking

The Social Perspective Taking Acts Measure (SPTAM) is a scenario-based instrument that analyzes a participant's written short answer responses to standard questions across social situations that commonly occur in middle schools. The SPTAM assesses the way individuals, at grade four and up, take multiple individuals' perspectives into account when responding to questions about hypothetical social dilemmas. As operationalized in the SPTAM, social perspective taking refers to the skills people can draw upon to “read” the social world, for example, through print text, through social discourse, and in the navigation of complex social relationships and civic participation. A coding manual is used to assess the expression of three performative social skills (acts): the acknowledgment of those parties relevant to the situation described in the scenario, the articulation of the thoughts and feelings of selected parties acknowledged therein, and the positioning of those scenario-based actors, depending upon their roles, circumstances, contexts, cultural background, and motivations. A validation study of the SPTAM found evidence of good internal reliability (alphas for the three scales, acknowledgment, articulation, and positioning were .80, .83, and .70, respectively), with construct validity confirmed by the findings that girls and older participants exhibited better performance than boys and younger students, and that the SPTAM exhibited a negative association with aggressive interpersonal strategies and small to moderate associations with academic language and with basic reading skills (Diazgranados, Selman, & Dionne, 2015). In the present study, we utilize only the articulation and positioning scales. Acknowledgment, though also important, captures a more basic perspective-taking competency while the articulation and positioning scales reflect more advanced perspective-taking skills needed by early adolescents to comprehend more complex texts.

Complex Reasoning

Based on King and Kitchener's (1994) work, Lectica, Inc. built a set of developmental assessments that measure the complexity of complex reasoning skills in children and adults, referred to collectively as reflective judgment (RFJ) tasks. The version used in this study is the RFJ001, which, like all forms of the RFJ, assesses the way individuals apply what they know about inquiry, evidence, truth, knowledge, reasoning, conflict, and deliberation to the solution of complex problems. This computer-administered assessment presents students with a scenario, typically a complex but familiar disagreement that has no correct answer. Students write responses to a series of questions designed to elicit strategies for deciding on an answer to the disagreement. Scores on the RFJ001, like scores for all of Lectica's assessments (Dawson & Stein, 2008, 2011), represent levels on Fischer's dynamic skill scale, a life-span scale of increasing hierarchical complexity (Fischer 1980, 2008; Fischer & Bidell, 2006). Responses are scored with low-inference rubrics in which each choice is associated with a phase (one-fourth of a developmental level) on the skill scale. Previous research has shown that the RFJ is sensitive to developmental differences, and reliably measures complex reasoning skills that are distinguishable from literacy skills (Dawson, 2014; Dawson & Stein, 2012). Based on the results of a confirmatory Rasch item response analysis of 3,754 performances spanning nine phases, Dawson (2014) has reported a person separation reliability of .91 with an estimated alpha of .94.

Deep Reading Comprehension

The Global Integrated Scenario-based Assessments (GISA) developed by ETS are computer-based assessments that use scenarios, technology, and reading strategies to motivate students, to model skilled reading, and to help disentangle key areas for improvement. In a scenario-based assessment design, students are given a plausible purpose for reading (e.g., decide if a wind farm is a good idea for your community) and a collection of source materials (e.g., blog, website, e-mail, news article, textbook excerpt).

Example scenarios include having students imagine that they are preparing to lead a class discussion or that they are part of a study group. Sources are written by a variety of authors that include a range of perspectives and levels of credibility on the issue in question. As such, students are expected to evaluate, integrate, and synthesize the materials in order to make a decision or solve the problem outlined in their original purpose for reading. Because the sources are thematic and the materials are sequenced to build up students' understanding of the topic, deeper questions can be asked of the student that not only tap their basic understanding (what the text is about), but also their ability to apply what they read to different contexts, situations, and perspectives.

Deep comprehension is in part facilitated by adding a social dimension to the assessment. In a typical GISA assessment, there are a number of simulated authority figures (e.g., teacher) and peers (e.g., students). The simulated teacher is designed to clarify expectations and provide hints that encourage desired responses. Simulated peers also may serve a similar function, but sometimes the test taker is asked to comment on the simulated peer's understanding of the sources that may contain misconceptions or errors in reasoning. Adding the simulated peer responses allows the test designer to more effectively target metacognitive and self-regulatory behaviors that encourage deeper understanding and perspective taking. This strategic approach to reading is also facilitated by adding a range of empirically supported reading strategies as tasks (e.g., summarization, use of graphic organizers, questioning). The reading strategies serve to encourage more thoughtful processing and model skilled reading.

GISA batteries cover a range of content areas, including science, social studies, and English language arts. Presented with a specific purpose for reading, students are then provided materials that are focused on a common topic and that include a variety of text types that students regularly encounter (e.g., expository texts, fiction, e-mail, web pages, and blogs). Students advance through the materials in a structured way that enables them to produce evidence of complex mental models of text content, organize what they read, and synthesize what they have learned to satisfy their original purpose for reading.

Although it is a relatively new instrument and more nuanced evaluations are ongoing, the GISA is appropriate for a wide range of ability levels and can reliably measure a range of “21st-century” reading skills that go beyond those assessed in more traditional, low-inference comprehension assessments (O'Reilly et al., 2014; Sabatini, O'Reilly, Halderman, & Bruce, 2014a, b). ETS reported results from a study of middle school students administered the organic farming version of the GISA that revealed adequate psychometric properties (Sabatini et al., 2014b), including internal consistency (alpha) reliability (0.89) and split half reliability (.76), with each half of the test showing adequate alpha reliability (α = 0.80 and α = 0.82, respectively). Test–retest reliability was r (283) = .87 and there was no significant difference in mean scores. The GISA also demonstrated a strong concurrent validity with more conventional reading comprehension tests, as well as component reading subtests. For this study, the following versions of the GISA were administered: Deserts (fourth and fifth grade, fall 2012), Satellites (fourth and fifth grade, spring 2013), Organic Farming (sixth and seventh grade, fall 2012) and Wind Power (sixth and seventh grade, spring 2013). Although the GISA is currently designed to produce a single score, the items and tasks in these forms in principle measure skills including evaluation of websites; distinguishing claims and evidence; integrating information across multiple texts; interpreting graphs or cartoons; questioning; predicting; summarizing and evaluating summaries; paraphrasing; using graphic organizers; analogical thinking; and learning. Scores are reported on a common, cross-form scale based on a large-scale study conducted by the ETS research team. Scaling used a concurrent, multigroup approach, with a two-parameter logistic IRT model (2PL) for form pairs.

Analysis Plan

Preliminary Analysis

Table 2 presents means, standard deviations, and correlation coefficients for the main variables of interest. Spring GISA scores (our measure of deep reading comprehension) were on average higher than, and were highly associated with, fall GISA scores, r = .72, p < .001. In addition, spring GISA scores were associated with the hypothesized predictors of GISA, including the two perspective-taking scores—perspective articulation (r = .38, p < .001) and positioning (r = .30, p < .001) as well as complex reasoning scores (r = .51, p < .001) and academic language (r = .68, p < .001). Predictors of reading comprehension were positively correlated with each other as well, r = .26–.64, p < .001.

Table 2. Means and standard deviations of the measures (N = 1965).

	M	SD	1	2	3	4	5	6
1. Spring GISA Score	976.87	69.21	—
2. Fall GISA Score	963.56	69.30	0.72	—
3. Fall Perspective Articulation	2.03	0.66	0.38	0.37	—
4. Fall Perspective Positioning	0.61	0.41	0.30	0.23	0.29	—
5. Fall Academic Language	0.30	1.04	0.68	0.67	0.46	0.36	—
6. Fall Complex Reasoning (Subsample n = 1610)	9.52	0.24	0.51	0.50	0.34	0.26	0.64

*** p < .001

Missing Information

The initial analytic sample for this study included students who attended the participating control schools during the 2012–2013 school year and took any of the assessments administered in this school year (n = 2,933). Among the participants, fewer than 3% were missing demographic information (race/ethnicity 3%; gender, free/reduced-price lunch eligibility, ELL status, receiving special education 1%). The rates of missingness were larger for the student assessment data, ranging from 12% to 25%. Specifically, GISA scores were not available from 17% of the participants in both fall and spring; the perspective-taking scores (15%) and CALS (12%) had slightly lower rates of missingness, whereas 25% of the complex reasoning scores were missing.

To maintain consistency across analytic models with respect to degrees of freedom and allow comparison between models, only participants with complete data on the main predictor variables (demographic characteristics, fall deep comprehension scores, perspective-taking skills, and academic language scores) were included in the primary models (n = 1,965). Compared to the full sample, this final sample was more likely to include White students (c² = 4.38, p < .05), compared to the full sample, and less likely to include students who were Black (c² = 6.61, p = .01), mixed/other race (c² = 6.49, p = .01), English language learners (c² = 10.45, p = .001), and eligible for special education (c² = 35.98, p < .001).

Because of the large proportion of missing information, complex reasoning was explored in a separate set of exploratory models, in which only the subsample with complete complex reasoning scores (n = 1,615) was included. Compared to the final sample above, this exploratory sample was even more likely to be White (c² = 34.98, p < .001) and Latino (c² = 3.49, p > .05), but less likely to include students who are Black (c² = 60.84, p < .001), eligible for free/reduced-price lunch (c² = 5.58, p < .05), or eligible for special education (c² = 17.38, p < .001)

The high rate of missingness in assessments is not unexpected, given that 10%–46% of students switch schools within one academic year in the participating schools, which primarily serve low-income communities. In addition to high student mobility, there is an impact of student absences on testing days.

Multilevel Models

Research questions were tested using a series of multilevel models to address the nested structure of the data (i.e., children nested in classrooms and schools) using STATA xtmixed procedure. In all models, continuous variables were group mean-centered within classrooms, because the main interest of the current study is to examine relations between individual students' academic-related skills and their reading comprehension performance. This approach allows unbiased estimates of relations between variables at the individual level and produces the most accurate estimates of the slope variance (Enders & Tofighi, 2007; Raudenbush & Bryk, 2002). Full maximum likelihood estimation was used in order to handle missing data in the outcome variable (Allison, 2012).

Because of the data limitation of the complex reasoning measure, we tested the research question using two separate sets of models. The first set of models examined the role of perspective-taking skills and academic language with the full analytic sample (n = 1,965). The second analysis explored the additional role of complex reasoning with a subset of the larger sample (n = 1,615).

Results

Prior to testing hypotheses, an unconditional (i.e., null) model (Table 3, model 1) was estimated to determine the amount of variance within and between classrooms and between schools in three-level models. The intraclass correlation coefficients (ICCs) of GISA scores were .14 at the school level and .14 at the classroom level. This suggested that the greatest variance in deep reading comprehension scores was at the individual student level (.72), whereas the variance between classrooms and schools was large enough to be worth examining (>.10, Lee, 2000), with design effects >2 for both classroom (2.92) and school level (2.24). Therefore we used three-level models for further analyses.

Table 3. Predictors of spring GISA scores: Role of academic language and perspective taking (N = 1965).

	Model 1		Model 2		Model 3		Model 4			Model 5
Variables	Coefficient	SE	Coefficient	SE	Coefficient	SE	Coefficient	SE		Coefficient	SE
Intercept	967.12	7.65	962.15	8.53	962.26	8.50	957.16	8.65		957.55	8.63
Grade 5			14.27	6.40	14.41	6.52	13.53	6.73		13.64	6.78
Grade 6			26.20	6.13	26.26	6.24	24.82	6.44		24.86	6.49
Grade 7			28.03	6.22	27.83	6.34	27.04	6.55		26.96	6.59
Female			−0.54	2.25	−3.61	2.30	−0.85	2.15		−2.41	2.21
White			4.16	3.86	4.33	3.82	2.82	3.69		3.00	3.68
Asian			12.47	6.88	12.82	6.82	9.63	6.58		10.01	6.56
Latino			4.97	3.66	5.57	3.62	4.69	3.49		5.07	3.49
Native American/Pacific Islander			−1.17	14.15	−1.79	14.01	1.08	13.50		0.36	13.46
Mixed/Other			18.66	11.13	18.48	11.02	16.00	10.63		16.10	10.59
Free/Reduced-Price Lunch			−11.93	3.32	−11.53	3.29	−8.36	3.18		−8.39	3.17
English Language Learners			−22.64	4.57	−20.67	4.54	−14.30	4.42		−13.83	4.41
Special Education			−25.55	3.50	−22.42	3.52	−16.30	3.43		−15.22	3.44
Fall GISA Scores			0.59	0.02	0.56	0.02	0.44	0.02		0.43	0.02
Fall Perspective Articulation					6.82	2.05				2.65	2.00
Fall Perspective Positioning					12.40	3.17				7.91	3.07
Fall Academic Language							20.66	1.66		19.54	1.71
Random effects	Variance		Variance		Variance		Variance			Variance
School Level	583.51		459.24		447.64		469.41			462.00
Classroom Level	612.64		374.38		396.46		442.68			451.69
Student Level	3,654.04		2,013.30		1,971.81		1,828.07			1,816.22

* p < .05;

** p < .01;

*** p < .001

Role of Perspective-Taking Skills and Academic Language

Models presented in Table 3 tested the role of the two perspective-taking skills (articulation and positioning) and academic language in predicting deep comprehension. Model 2 tested whether demographic and other student characteristics predict spring GISA scores, when controlling for fall scores. The findings suggest that fifth (coefficient = 14.27, p < .05), sixth (coefficient = 26.20, p < .001), and seventh (coefficient = 28.03, p < .001) graders had steeper growth than fourth graders from fall to spring on deep comprehension. We found no difference by students' racial and ethnic background or gender. However, students who were low income (coefficient = −11.93, p < .001), ELLs (coefficient = −22.64, p < .001), and receiving special education (coefficient = −25.55, p < .001) had slower growth in their deep comprehension skills across one academic year than their peers. These student characteristics explained 45% of the variance in deep reading comprehension at the student level (c² = 1024.20, p < .001).

Model 3 examined the role of the two perspective-taking measures, perspective articulation and positioning. Controlling for student covariates, both perspective articulation (coefficient = 6.82, p < .001) and positioning (coefficient = 12.40, p < .001) measured in fall positively predicted the spring deep reading comprehension scores. This model explained an additional 1% of the total variance at the student level (Model 2 versus 3, c² = 30.40, p < .001).

The role of academic language is tested in Model 4. Fall academic language skill was also a strong and positive predictor of spring deep comprehension scores (coefficient = 20.66, p < .001). Inclusion of academic language in the model explained an additional 5% of the total variance at the student level over and above the student covariates (Model 2 versus 4, c² = 146.28, p < .001).

Finally, Model 5 tested the role of both perspective taking skills and academic language. Fall perspective positioning skill (coefficient = 7.91, p < .01) positively predicted spring deep comprehension scores over and above the fall academic language skills and student covariates. Academic language skills also positively predicted deep comprehension (coefficient = 19.54, p < .001). This final model explained 50% of the total variance in deep comprehension (versus Model 1, c² = 1179.76, p < .001), explaining an additional 5% compared to Model 2 (c² = 155.56, p < .001), 4% compared to Model 3 (c² = 125.16, p < .001), and 0.3% of the total variance than Model 4 (c² = 9.28, p < .01).

Exploring the Role of Complex Reasoning

Due to the constraints on the available data, we examined the role of complex reasoning in predicting GISA in a separate set of models (Table 4). Model 1 presents the null model for the subsample with available complex reasoning scores. This subsample had slightly smaller ICCs on deep reading comprehension, .11 at the school level and .12 at the classroom level.

Table 4. Predictors of spring GISA scores: Role of complex reasoning (N = 1615).

	Model 1		Model 2		Model 3		Model 4		Model 5
Variables	Coefficient	SE	Coefficient	SE	Coefficient	SE	Coefficient	SE	Coefficient	SE
Intercept	969.83	7.62	961.77	9.07	960.75	9.07	956.85	9.17	956.86	9.16
Grade 5			12.99^+	7.20	11.95^+	7.22	12.07	7.60	11.73	7.58
Grade 6			26.38	6.77	26.13	6.79	24.80	7.14	24.80	7.13
Grade 7			28.50	6.86	27.62	6.88	27.79	7.24	27.48	7.23
Female			−2.72	2.50	−3.99	2.47	−3.89	2.44	−4.36^+	2.44
White			6.61	4.24	6.99^+	4.18	4.56	4.01	4.89	4.01
Asian			13.61^+	7.57	11.84	7.45	9.88	7.16	9.49	7.14
Latino			7.54^+	4.06	7.50^+	4.00	6.15	3.85	6.27	3.84
Native American/Pacific Islander			−6.96	15.10	−4.56	14.87	−1.10	14.27	−0.65	14.23
Mixed/Other			14.12	11.67	13.95	11.48	12.33	11.01	12.42	10.99
Free/Reduced-Price Lunch			−10.98	3.63	−9.66	3.58	−7.10	3.44	−6.88	3.44
English Language Learners			−20.22	5.22	−17.04	5.17	−10.25	5.01	−9.75^+	5.00
Special Education			−25.06	3.99	−22.19	3.95	−14.18	3.86	−13.86	3.86
Fall GISA Scores			0.60	0.02	0.54	0.02	0.43	0.02	0.42	0.02
Fall Perspective Articulation							1.43	2.19	1.22	2.18
Fall Perspective Positioning							8.69	3.35	8.49	3.34
Fall Academic Language							21.90	1.91	20.13	2.02
Fall Complex Reasoning					44.24	6.68			18.01	6.83
Random effects	Variance		Variance		Variance		Variance		Variance
School Level	560.59		456.62		459.47		460.00		460.17
Classroom Level	597.49		386.76		402.82		477.75		475.41
Student Level	3,758.70		2,066.31		2,000.17		1,831.51		1,822.72

* p < .05;

** p < .01;

*** p < .001

Model 2 including the student covariates produced findings similar to those for the full sample, with sixth (coefficient = 26.38, p < .001) and seventh (coefficient = 28.50, p < .001) grade showing significantly steeper growth across one academic year than fourth graders, and fifth graders showing a trend toward steeper growth (coefficient = 12.99, p = .07). Students who were low income (coefficient = −10.98, p < .01), ELLs (coefficient = −20.22, p < .001), and in special education (coefficient = −25.06, p < .001) had slower growth in their deep comprehension skills than their peers in the subsample, echoing the findings for the full sample. These student characteristics explained 45% of the total variance on GISA scores at the student level (c² = 853.73, p < .001).

The role of complex reasoning is tested in Model 3. The results suggest that fall scores were a strong and positive predictor of spring deep comprehension scores (coefficient = 44.24, p < .001). Inclusion of complex reasoning scores in the model explained an additional 2% of the total variance at the student level over and above the student covariates (Model 2 versus 3, c² = 43.17, p > .001).

Finally, Model 5 examined whether complex reasoning contributes to explaining the growth of deep reading comprehension over and above perspective taking and academic language skills (Model 4). The findings reveal that fall complex reasoning independently and positively predicts deep reading comprehension (coefficient = 18.01, p < .01), when perspective taking and academic language skills are controlled for. Including complex reasoning scores in the model explained an additional .02% of the total variance compared to Model 4 (Model 4 versus Model 5, c² = 6.94, p < .01). Overall, this final model including all three predictors of reading comprehension (perspective taking, academic language, and complex reasoning) and student covariates explained 52% of the total variance in deep comprehension at the student level (versus Model 1, c² = 1012.84, p < .001).

Discussion

The goal of the study presented here was to provide a first test of the hypothesis that deep reading comprehension is codetermined by students' ability to understand academic language, to take and understand social perspectives, and to engage in complex reasoning about challenging problems. The findings support the credibility of the hypothesis. Academic language was the strongest of our hypothesized predictors of deep comprehension, suggesting that this is an important area of focus to prepare students for secondary school texts with their increasingly unfamiliar and challenging language (e.g., Schleppegrell et al., 2004; Schleppegrell & de Oliveira, 2006). Both perspective articulation and positioning were found to have relationships with deep comprehension, although when academic language is added to the model, only positioning is statistically significant, suggesting that, in addition to academic language, it is particularly important for early adolescents to be able to interpret varied perspectives in the context of a character's or author's roles, circumstances, cultural backgrounds, etc. This is aligned with Kessler and Donahue (2008)'s work suggesting that early adolescents' comprehension can be limited by insufficient skills for processing characters' multiple and sometimes conflicting positions.

Additional exploratory analyses from the present study indicate that complex reasoning is also a significant predictor of deep comprehension. Previous research has shown complex reasoning to be related to academic and verbal ability (Wood, 1997), but we believe this is the first study to demonstrate that complex reasoning predicts deep comprehension. Although these analyses were conducted with a reduced sample and are interpreted with caution, they do offer preliminary evidence that complex reasoning may be important for the reading comprehension tasks required of young adolescents. In addition, academic language and perspective positioning continue to be significant predictors of deep comprehension in these exploratory analyses that include complex reasoning.

Taken together, these findings on the roles of academic language, perspective taking, and complex reasoning in deep comprehension suggest that, for students in grades four through seven, we need to consider other models of reading comprehension beyond the SVR. In particular, as students enter the middle grades and need to understand increasingly complex texts, decoding and oral language skills may be necessary but insufficient for reading comprehension. Indeed the literature in support of the SVR has most often focused on younger students, and others have argued that the SVR neglects other skills, such as efficiency and fluency, that may become more important for students as they reach the higher grades (Høien-Tengesdal & Høien, 2012; Macaruso & Shankweiler, 2010; Silverman et al., 2013). In addition, research on the SVR has utilized measures of reading comprehension that assess a relatively basic level of reading comprehension using relatively simple texts and noninferential questions, while success in middle and high school requires students to learn from more complex tests across disciplines and to interpret and analyze these texts in more sophisticated ways. Our study is one of the first to employ a “deep comprehension” assessment that incorporates a variety of textual sources and more challenging reading comprehension tasks, such as evaluating and synthesizing materials and applying what has been read to different contexts and perspectives. Thus, the results of this study demonstrate not only that other reading-related skills (e.g., academic language) are important in early adolescence, but also that such skills are important for the deep comprehension required of students in upper-grade classrooms.

This is, of course, only the first step in the analytic program testing this hypothesis; future analyses should examine whether these three components predict reading comprehension beyond the contributions of decoding and oral language. In future work we will conduct similar analyses for a subsample of participants for whom we have measures of general vocabulary and decoding, as well as fluency, and more traditional assessments of reading comprehension, including state ELA tests, allowing us to more directly compare the SVR with our model of deep reading comprehension predicted by academic language, perspective taking, and complex reasoning.

Our analysis of predictors of deep comprehension also included several key demographic factors that deserve mention. After controlling fall deep reading comprehension scores, several demographic variables predicted spring comprehension scores. Students who are eligible for free or reduced-price lunch, are English language learners, and/or have a special education classification scored significantly lower on deep reading comprehension. Although racial/ethnic differences on academic outcomes including reading comprehension are common (NCES, 2013; Reardon, Valentino, & Shores, 2012), there was no relationship between students' racial/ethnic backgrounds and GISA comprehension scores. This is true for other recent studies using the GISA (O'Reilly et al., 2015). Finally, there was a significant effect for grade, with students in higher grades scoring significantly higher than fourth-grade students on deep comprehension.

There are of course limitations to the analysis presented here. Missing data are a chronic problem in school-based research, one that is exacerbated when the data derive from four distinct assessments, administered in different sessions, in whole-class configurations, and in two cases digitally in elderly computer labs. Although we made every effort to reschedule testing sessions when absences were a significant problem, high school mobility rates and shifting class schedules rendered success incomplete.

Nonetheless, we are able to report findings based on a substantial number of students. Unfortunately, there is some suggestion in the pattern of missingness that the students whose scores were omitted differed on demographic factors from those included in the final analytic sample or subsample, a finding that requires ongoing analysis. Fortunately, because the predictors reported here were assessed in only one wave of data collection, in a study in which six waves are available on some participants and four on most, we will have the opportunity in future analyses to pursue these same questions with additional waves of data, modeling growth trajectories over as many as three years for both predictors and the outcome.

Another limitation is the lack of classroom and school-level predictors (e.g., aggregated student demographics) despite sufficient classroom and school-level variance in deep comprehension to warrant multilevel modeling. We decided not to include such predictors in this analysis because of the challenges of multicollinearity among classroom and school-level demographics and because our focus in this paper is on student-level questions. Future analyses will examine the context-level variation in deep comprehension.

Despite these limitations, these results provide preliminary evidence that academic language, perspective taking, and complex reasoning predict deep reading comprehension. This finding has important implications for theories of reading comprehension and for designing interventions to improve comprehension outcomes. These three predictive factors have not previously been systematically attended to in curricular design or instruction. Programmatic efforts to manipulate these input factors through curricular design and/or special interventions and to assess impacts on reading comprehension outcomes will provide the ultimate test of their importance as components of the deep comprehension process.

The relevance of academic language, perspective taking, and complex reasoning to deep comprehension suggests as well the importance of ensuring that teachers of literacy-heavy content areas (history and science, as well as English language arts) understand these phenomena. It is difficult to predict where students will struggle with the text or misinterpret author's intent without some understanding of academic language and perspective taking. Texts encountered in history are inherently perspectival, and science texts demand following extended lines of reasoning explained verbally. Teachers' sensitivity to these challenges will improve content-area learning as well as literacy outcomes.

Practices widely promoted as helping students reach the expectations of the Common Core State Standards include close reading and assigning more complex texts. Our findings suggest that these practices are unlikely by themselves to be helpful to students struggling with academic language, perspective taking, and complex reasoning, and might in fact lead to frustration and reduced motivation. A better understanding of the processes underlying deep reading comprehension will, we hope, generate approaches to instruction that directly address the linguistic and cognitive challenges students face.

Acknowledgments

The authors contributed to this manuscript in various ways. LaRusso and Jones directed the research and advised on the analytic strategy. Kim conducted the analyses. Selman, Uccelli, and Dawson developed assessments for the key predictors, perspective taking, academic language, and complex reasoning, respectively. The manuscript was prepared primarily by Snow, LaRusso, and Kim. Snow and Donovan were co-PIs. We would like to thank James Kim, Lowry Hemphill, McCaila Ingold-Smith, Jen Winsor, Jill Joseph, Chris Barr, Emily Phillips Galloway, Silvia Diazgranados, and our many research assistants for their contributions to this research, as well as the students, teachers, and administrators for their participation.

Funding

The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305F100026 to the Strategic Educational Research Partnership Institute and grant R305F100005 to Educational Testing Service as part of the Reading for Understanding Research Initiative. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education.