Validation of sub-constructs in reading comprehension tests using teachers’ classification of cognitive targets

ABSTRACT

Reading comprehension is often treated as a multidimensional construct. In many reading tests, items are distributed over reading process categories to represent the subskills expected to constitute comprehension. This study explores (a) the extent to which specified subskills of reading comprehension tests are conceptually conceivable to teachers, who score and use national reading test results and (b) the extent to which teachers agree on how to locate and define item difficulty in terms of expected text comprehension. Eleven teachers of Swedish were asked to classify items from a national reading test in Sweden by process categories similar to the categories used in the PIRLS reading test. They were also asked to describe the type of comprehension necessary for solving the items. Findings of the study suggest that the reliability of item classification is limited and that teachers’ perception of item difficulty is diverse. Although the data set in the study is limited, the findings indicate, in line with recent validity theory, that the division of reading comprehension into subskills by cognitive process level will require further validity evidence and should be treated with caution. Implications for the interpretation of test scores and for test development are discussed.

Introduction

According to Messick’s (1989) classic definition, validity refers to “the degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness of inferences and actions on the basis of test scores or other modes of assessment” (p. 13, italics in original). Argument for test validity, then, following Kane (2013), must be based on an evaluation of the plausibility of the proposed interpretations and uses of test scores. In national test systems, like the ones found in the Nordic countries, classroom teachers all over the country, instead of small panels of experts, take responsibility both for the scoring of tests and for the interpretation and use of test results in the classroom (EACEA, 2009). Therefore, in these systems, teacher interpretations of test content and score meaning provide vital evidence for test validity.

This article concerns two issues of importance to the development and use of reading comprehension tests: the validation of sub-constructs of the test and the consistency in perceiving the difficulty of individual test items. The article reports an empirical study of (a) the extent to which specified sub-constructs are conceptually conceivable to teachers, who score and use national reading test results, and (b) the extent to which teachers agree on how to locate and define item difficulty in terms of expected text comprehension. In this way the article aims to contribute relevant knowledge about validity in the assessment of reading comprehension and to discuss ways of ensuring construct and content representativeness in tests.

Reading comprehension is commonly conceived of as a multidimensional form of processing, as opposed to being a unitary construct (Campbell, 2005; OECD, 2009; van den Broek, 2012). This suggests that the reading situation, text, topic, and reading purposes will affect the nature of reading in such a way that when measuring reading proficiency, the selection of reading material, the assumed reading purposes, and also item construction will have a significant impact on students’ test results. This has been verified in many empirical studies (Best, Ozuru, Floyd., & McNamara, 2006; Francis et al., 2006; Keenan, Betjemann, & Olson, 2008; van den Bergh, 1990). Moreover, for the last couple of decades it has been fashionable in the construction of standardized tests of reading to categorize comprehension components by taxonomical models of cognitive complexity, often going from locating explicitly mentioned pieces of information, to inferencing, interpreting, and critically evaluating text (Pearson & Hamm, 2005; Song, 2008). These are the categories, or reading processes, defined in, for instance, the PISA reading test (OECD, 2009). Similar categories are used in the PIRLS test (IEA, 2009) as well as in many national reading tests (Tengberg, 2017).

However, while reading processes may be identified conceptually (i.e., perceived as psychological multidimensionality, Henning, 1992), evidence of psychometric divisibility of reading comprehension has been scarce. Although many researchers have suggested classifications of different dimensions (Davis, 1968; Khalifa & Weir, 2009; Langer, 1995; McNamara & Magliano, 2009), there is still little consensus in the field as to how reading comprehension processes might validly be classified, or even if separable subskills of comprehension really exist at all (Alderson, 2000; Rupp, 2012). Several empirical studies, using both analyses of intercorrelations between proposed subskills and factor analyses, have demonstrated the difficulty of separating reading comprehension into reliable factors or sub-components (Meijer & van Gelderen, 2002; Rost, 1993; Schedl, Gordon, Carey, & Tang, 1996; Song, 2008; Spearritt, 1972; van Steensel, Oostdam, & van Gelderen, 2012).

Another way of validating a multidimensional view of reading comprehension involves studying the cognitive processes activated by test takers while solving tasks of different categories. In an experimental study, Rouet, Vidal-Abarca, Erboul, and Millogo (2001) found that undergraduate students presented with a scientific text activated a “review-and-integrate search pattern” when answering high-level questions, and a “locate-and-memorize pattern” when answering low-level questions.¹ Field (2013), among others, has argued that validation of the cognitive processes underlying language test performance, concerning both listening and reading comprehension, is an area where much more research is needed. Cognitive validation means ensuring, first of all, that the processes activated during test-taking resemble the cognitive processing that test developers actually aim to assess, and second, that the processes elicited by test items are representative for the processes activated and needed during non-test reading (Khalifa & Weir, 2009; Pearson & Hamm, 2005).

In cases where teachers of reading are expected to make formative use of reading test results, as in the Nordic countries, cognitive validation of test results also entails that teachers interpret cognitive complexity and difficulty of test items relatively consistently. This is especially important when tests are meant to measure comprehension according to curriculum goals, which is the case for national reading tests in the Nordic countries (Tengberg, 2017). In other words, teachers should be able to reach a shared understanding of the cognitive demands offered by test items. To validate the test by ensuring reliable classification of items will then work as a prerequisite for common standards in teachers’ feedback on student performance. If strengths and weaknesses of student performance are not identified correctly, then the information provided by test scores will have less valid implications for future reading instruction. The test design may thus fail to support student progress, something that inevitably will have the most damaging effect on the weakest performing readers (i.e., those who are in the direst need of adequate instructional support).

A framework of reading processes that has gained a great deal of attention in the recent decade is the threefold division into “aspects of reading” used in the PISA reading test. Originally, five aspects of reading were defined, but because reporting on five separate subscales would require a much more extensive collection of items, the framework was organized into three broader categories: access and retrieve; integrate and interpret; and reflect and evaluate (OECD, 2009). In the original PISA framework (OECD, 1999), the distinguishing characteristics of the different aspects were defined. First, a distinction is made between items requiring the reader to use information primarily from within the text and items that require the reader to draw upon outside knowledge. A second distinction concerns whether the reader must focus on the text as a whole or focus on specific parts of the text. A third distinguishing characteristic concerns whether the reader is asked to relate different pieces or parts of information to each other or to focus on parts of information in isolation. Finally, a fourth distinction is made between items where the reader should focus on the content of the text and items that deal with text structure (OECD, 1999, p. 29).

OECD admits that the definitions and the distinguishing markers necessarily oversimplify the complexity of each aspect. In addition, the specificity of tasks and the difficulty, in terms of required text comprehension, inevitably relates to the way information is structured in the text (Cerdan, Vidal-Abarca, Martinez, Gilabert, & Gil, 2009; Lumley, Routitsky, Mendelovits, & Ramalingam, 2012; Mosenthal, 1996; Rouet et al., 2001). Semantic relationship between propositions, the embeddedness of propositions, and the range and type of match between question formulation and text available to the reader, are all qualities that affect the way the cognitive load of items is perceived (Mosenthal, 1996). In addition, the level and the type of difficulty of items will be affected by readers’ familiarity with the topic, the genre and the format of the text (Alderson, 2000), and the depth of understanding or the extent of textual support requested by the scoring rubric (Solheim & Skaftun, 2009).

These complexities are important to analyze. Because, as Song (2008) points out,

Although the concept of describable subskills is highly controversial, in practice, it is commonplace for language teachers and language test developers to distinguish different comprehension subskills or levels of understanding of a given text as a basis for planning a syllabus, describing students’ language competence, and developing test items in a mother tongue as well as in a foreign language (Song, 2008, p. 436).

To summarize, cognitive validation of reading comprehension testing requires not only that suggested cognitive dimensions are psychometrically distinct from each other but also that they are conceivable as psychological dimensions (Henning, 1992; Pearson & Hamm, 2005). Reliable reporting by subscales necessitates psychometric evidence that the subskills exist, but for such results to bear meaning in the classroom, experienced reading teachers should be able to identify items in terms of the suggested reading process categories. If teachers’ classification of reading test items, which are aimed to measure reading comprehension according to curriculum goals, is not reliable, it will pose a threat to test validity in two different ways. First, it may be an indication that the items in question do not measure the intended skills according to the given framework. Second, it may be an indication that there is a lack of alignment between how subskills are defined in the test and how they are used and taught in the classroom. In both cases the interpretation of test scores will be undermined by a lack of validity evidence.

In alignment studies, where ratings of cognitive demand are used to evaluate items against standards, agreement between subject matter experts is essential. Herman, Webb, and Zuniga (2007), for instance, investigated the reliability of such alignment indices and noted considerable variability among high school Mathematics teachers in judging item dimensionality and depth of knowledge.² In a similar vein, Lombardi, Seburn, Conley, and Snow (2010) found higher reliability for cognitive demand ratings of Mathematics items than for English/Language Arts items, and higher reliability for ratings of cognitive demand than for rigor (level of difficulty) of items.

For the cognitive processes of reading comprehension items, previous empirical studies have shown that classifying items according to a reading framework is difficult even for trained reading researchers. In 1992 the L1 reading section of the American National Assessment of Educational Progress (NAEP) included a framework of four different reading processes, or stances (Pearson & Hamm, 2005).³ After administration, a study was conducted to analyze the alignment between items and framework. According to Glaser and Linn (1994), the results showed that the fit was moderate (only 50% agreement) and that the categories were judged not to be discrete from one another (see also DeStefano, Pearson, & Afflerbach, 1997). In an earlier study Alderson and Lukmani (1989) presented nine university teachers with an L2 reading test and let them rate whether the items were targeting “lower,” “middle,” or “higher” order abilities. The analysis showed that for a majority of the items (27 of 41) there was very little agreement on classification between judges. Similarly, when judges were asked to describe what they thought each item tested, their account often varied considerably. Looking further at the items where the judges actually did reach some agreement, the researchers found little relationship between these items either in terms of difficulty or in terms of process level.

However, the suggestion that judges would be unable to reach agreement on subskills tested by particular items has been challenged in later studies (Bachman, Davidson, & Milanovic, 1996; Brutten, Perkins, & Upshur, 1991; Lumley, 1993). For instance, Lumley (1993) let a group of five raters judge the difficulty of subskills and match subskills to individual items in an EAP test for students of non-English-speaking background. The results showed high level of agreement between raters and thus supported the validity and, in Lumley’s words, the usefulness of the construct (p. 230). It is worth noting that these studies all concern subskills in L2 reading tests and might not be generalizable to the perception of cognitive dimensions in L1 reading tests. In addition, the raters used in these studies were few and trained experts of the constructs, usually being test constructors themselves. To the extent that reading test construction and results are expected to influence both the content of classroom teaching and teachers’ semester grading, the cognitive validation should entail that tested subskills, or cognitive dimensions, are conceivable not only to testing experts but also to classroom teachers.

Context of the study

The national reading test in Swedish and Swedish as a second language⁴ in ninth grade (students aged 15–16) aims at measuring reading comprehension according to curriculum goals (Swedish National Agency for Education, 2011). It is based on a reading framework including four reading process levels (see Table 1). The process categories were introduced by inspiration primarily from the PIRLS test and serve to organize the item sample and to ensure a balanced coverage of the intended construct (Swedish National Agency for Education, 2014).⁵ However, while both the PIRLS and PISA frameworks provide extended descriptions of the process categories involved along with examples of items that might be used to assess them, respectively, (IEA, 2009; p. 23ff; OECD, 2009; p. 45ff), the descriptors of the Swedish framework are much briefer (see Table 1).

Table 1. Reading Process Categories With Descriptors.

Reading Process	Descriptor
1. Retrieve explicitly stated information	Retrieving explicitly stated information means locating and using information that is evident and stated in the text. Items testing this aspect of comprehension produce answers that can be judged as either correct or incorrect and treat the content of the text on a surface level.
2. Make straightforward inferences	Another way of understanding text is to make straightforward inferences. This means that the reader connect pieces of information locally and reformulate them into a response.
3. Integrate and interpret information and ideas, and reflect	In order to integrate and interpret information and ideas, and reflect, implied messages or latent meaning must be comprehended and retrieved. Items that require students to interpret/reflect usually cannot be judged in terms of correct or incorrect, but the interpretation/reflection can be more or less accurate with reference to the text.
4. Examine and evaluate content, language, and textual elements	Finally, a reader may step out of the text and examine and evaluate content, language, and textual elements. Examples of this aspect of comprehension are to perceive that a given text is an article or to compare information gathered from the text with information taken from previous knowledge.

Source: (Swedish National Agency for Education, 2014, p. 20)

In the Swedish test, student results are not reported by subscales, but to receive a passing grade (an E on the Swedish grading scale), students need to score points in at least three of the four process categories. In this way the process categories have a high-stakes function in the test, which is why cognitive validity of the framework is essential.

In Sweden, ninth grade is the final year of compulsory school. The national reading test is usually administered in March and serves the purpose of assessing student achievement according to curriculum goals and monitoring school performance. It is not an examination that decides the students’ final grades, but it is regarded as an important piece of evidence in assessing students’ progress according to the curriculum goals. A test result indicating that a student underachieves in some respect may, therefore, be put to formative use by the teacher in the period from March to June to help the student improve and thereby qualify for a higher final grade in Swedish. The way knowledge and skills are defined by national tests might also impact teachers’ general understanding of the skills they teach and certainly as they prepare students for taking the national tests (Koretz, 2008; Shohamy, Donitsa-Schmidt, & Ferman, 1996; Stobart, 2008). The cognitive validity of the test is, therefore, of imminent importance both to the students’ learning progress and to their final grades.

Research questions

Reliable classification of what different items actually test is crucial to the cognitive validity of any test design assumed to measure ability along some scheme of different subskills. To contribute relevant knowledge about the validity of cognitive process schemes in reading comprehension tests, this article addresses two distinct but interrelated research questions. First, to what extent are teachers able to classify reading test items reliably by reading process levels? Second, to what extent do teachers agree on how to define the difficulty of individual items in terms of expected text comprehension?

Method

Participants

The participants of the study were 12 qualified secondary school (Grades 6‒9) teachers of Swedish (all female), from 7 different schools in both rural and urban areas. Teaching experience ranged between 7 and 21 years and the teachers had all been administering and scoring national test in Swedish annually during their whole working career. In Sweden, students receive semester grades in each subject, meaning that the participants were also well experienced at assessing student achievements according to curricular goals. All of them volunteered by responding to an open invitation. The data collection was conducted during a one-day in-service workshop on assessment. One participant, however, repeatedly assigned items into several categories, against the explicit instruction given, which is why the final data set contains 11 raters.⁶

Procedure

The teachers were presented with 4 texts and 19 test items sampled from test material in use from 2013 to 2014. A week in advance, they were emailed pdf versions of the texts and asked to read them carefully a couple of times. During the workshop they were provided with the test constructors’ definitions of the four processes (see Table 1) and asked to classify items accordingly. Each item was, thus, classified by all teachers. Having administered the national tests several times, the teachers were familiar with the test design and most of them had read the texts and seen the items previously. It was expected, however, that they would not remember the way in which particular items were classified by cognitive processes.

In addition to classification, the teachers were also asked to write an open-ended description for each item of what they believed the student needed to understand specifically in the text to be able to answer the question. These descriptions were produced together with the classifications. In the data collection protocol each item was thus followed by an instruction (“This is what the student needs to understand in the text to be able to answer the question”) and space for an open-ended response.

Data Analysis

The data were analyzed in several steps. First, the classifications were plotted in a table (Rater x Item) for an initial review of agreement for each item. Data were then restructured (Item x Process) and Fleiss’s kappa for agreement between raters was computed by using the software package R (all calculations were double-checked in Excel). Fleiss’s kappa is the recommended measure for assessing overall reliability between multiple raters on categorical (nominal) data (Gwet, 2008; Landis & Koch, 1977). While Cohen’s kappa measures the agreement between two raters, Fleiss’s kappa can be extended to measure agreement between multiple raters. As with Cohen’s kappa, Fleiss’s kappa controls for the agreement expected by chance alone and uses no weighting. To interpret values of kappa, Landis and Koch proposed that 0–0.20 indicates slight agreement, 0.21–0.40 fair agreement, 0.41–0.60 moderate agreement, 0.61–0.80 substantial agreement, and 0.81–1.00 indicates almost perfect or perfect agreement. These guidelines, however, have been criticized and Krippendorff (1980), for instance, has suggested a more conservative guide for interpretation of kappa values, in which observed agreements below 0.67 should be disregarded, values between 0.67 and 0.80 should be seen as grounds for tentative conclusions, and that definite conclusions be made only for values above 0.80. As noted by Hallgren (2012), however, all interpretation of reliability estimates will be dependent on both the method of study and the research question. Furthermore, it is essential to bear in mind that some scales and distributions are easier to judge consistently than others. Thus, most reliability estimates need to be analyzed for the particular performances being assessed and the particular scale used.

When analyzing how the teachers locate and define difficulty in items, all written accounts were charted and coded according to the type of textual references that they made and according to the cognitive processes that they thought was required. The coding scheme was thus developed during the analysis and both categories and coding were recurrently re-assessed to eventually fit all teacher accounts into the final version of the coding scheme. Results of the coding have not been subjected to kappa analysis. This is partly because the reports included some missing data, to which Fleiss’s kappa is highly sensitive. The main reason, however, is that the purpose of coding is not to display statistics of item difficulty definitions but rather to structure the teacher accounts into meaningful groups to reveal topical patterns in how reading test items are conceived by those who score the tests. It needs to be stressed that this part of the analysis is not primarily focused on what the teachers believe the challenge of an item is but how they describe it or, more precisely, how they locate and define the type of comprehension necessary for successfully solving the item.

Results

The logic of the study is that if teachers, whose task it is to score the tests and use the results, can classify test items consistently, it will contribute to the validation of the reading framework by showing that the reading processes are conceptually conceivable to professionals in the field. If the teachers are not able to do this, then it may be seen as a threat to test validity and may also indicate that the transferability into formative use of the test results is endangered.

Classification of items by reading process categories

The teachers’ classifications of the 19 items are displayed in Table 2, together with an account of the official classification by the test constructor. As seen in the table, there are items (e.g., i14) for which all raters agree. They have all put i14 in category 4, in alignment with the test constructor. Similarly, 10 of the 11 raters have put i10 in the official category. Both of these items deal with conceptual knowledge. In i14 students are asked to assign genre to the input text. But if we look at i11, the test constructor classifies it as belonging to category 3, which none of the teachers do. For i18 all teachers agree on choosing category 2 while the test constructor defines it as a 3. There are also items where the variation is total in the sense that all four categories are suggested (see i6 and i17).

Table 2. Raters’ Classification of Items Into Reading Process Categories.

Item (i)	R1	R2	R3	R4	R5	R6	R7	R8	R9	R10	R11	Official
i1	2	3	3	2	3	3	3	2	3	3	3	3
i2	4	2	2	1	2	2	2	1	2	2	2	4
i3	1	2	1	1	1	2	2	1	2	1	1	1
i4	1	1	2	2	2	2	2	1	1	1	1	1
i5	3	1	3	3	2	2	3	2	3	2	1	2
i6	4	4	1	3	2	2	4	3	4	4	4	4
i7	3	3	3	2	3	3	3	2	3	2	3	3
i8	4	4	3	1	4	3	3	3	4	3	3	3
i9	3	3	3	3	3	3	3	2	3	3	2	3
i10	1	2	1	1	1	1	1	1	1	1	1	1
i11	2	1	2	1	2	2	2	1	2	2	2	3
i12	3	1	3	2	2	2	2	1	2	1	1	2
i13	2	2	3	2	2	2	2	2	3	2	2	2
i14	4	4	4	4	4	4	4	4	4	4	4	4
i15	2	2	2	2	2	3	2	3	1	3	3	4
i16	3	2	2	3	2	3	3	3	3	2	3	2
i17	4	1	2	3	1	1	3	2	4	2	2	2
i18	2	2	2	2	2	2	2	2	2	2	2	3
i19	4	1	4	4	1	4	1	4	1	2	1	4

To evaluate the overall agreement statistically within the group of raters, Fleiss’s kappa was calculated and is reported in Table 3. The results indicate what Landis and Koch (1977) would define as fair level of agreement between raters (κ = .35 (95% CI, .31 to .38), p < .0001). In practice this means that there is a considerable variation in how teachers classify the items according to reading process category. Table 3 also shows that there is significantly higher agreement when classifying items as category 4 than when using the other three categories.

Table 3. Fleiss’s Kappa for Interrater Agreement Between the 11 Raters.

alpha = 0.05	Total	Reading Process 1	Reading Process 2	Reading Process 3	Reading Process 4
Kappa	0.35	0.32	0.25	0.37	0.54
s.e.	0.02	0.03	0.03	0.03	0.03
p-value	0.00	0.00	0.00	0.00	0.00
Lower	0.31	0.26	0.19	0.31	0.47
Upper	0.38	0.38	0.31	0.43	0.60

The Fleiss’s kappa analysis also yields information about the proportion of pairs of raters that agree in their classification on each item (p_i). For five of the 19 items (i9; i10; i13; i14; i18) there was fairly high interrater agreement (p_i > .65), while for four items (i5; i6; i12; i17) interrater agreement was fairly low (p_i < .35). Studying these items separately, no association was found between the level of agreement and reading process category. However, four of five items with high interrater agreement were multiple choice (MC), while three of four items with low interrater agreement were constructed response (CR). The average p_i was also larger for MC items (.58) than for CR items (.42). According to an independent t-test the difference is not significant for the given sample (t (17) = −1.60, p = .13), but it may certainly be argued that the number of items is too small and that a larger sample might have detected a significant difference.

The principal finding in this section is that even teachers with extensive experience of scoring national tests (although not being trained experts) find it difficult to reach agreement about classification of reading test items into cognitive process categories. One way of exploring further why this is so difficult is to analyze how teachers locate and describe the particular challenge or difficulty in items, by using their own words rather than fixed labels.

Defining what students need to understand to solve individual items

Thus, in addition to classification, the teachers were also asked to describe in their own words what the students need to understand to solve the given items. These reports were then analyzed and grouped into different categories, depending on the type of references they made in explaining item difficulty (Table 4). It is worth repeating that the coding is not primarily focused on what they believe the challenge is but rather on how they locate and define difficulty of items. The coding and the table should thus be understood as displaying the points of reference teachers make when they describe what it is more precisely that a test taker must comprehend to solve the particular task.

Table 4. Type of References When Locating and Defining Item Difficulty.

1) Referring to reading process (e.g., according to process scheme) that should be used.

2) Referring to or cites places or aspects of the text where the answer can be found.

3) Referring to specific interpretations/inferences that the reader needs to make in order to come up with an adequate item response.

4) Referring to words or expressions (either in the text or in the item instruction) which the reader needs to understand in order to be able to respond.

5) Referring to comprehension of characters’ or main persons’ psychological background or motives.

6) Referring to necessary knowledge about formal aspects of the text.

A first impression of the accounts is that the majority of them are quite short (20 words or less). Still, they appear to represent an honest attempt to characterize the primary challenge of each item as precisely as possible. Naturally, there can be no single correct definition of what is necessary for a student to understand in a given text (or in a given test item). Rather, the categories suggested in Table 4 point to the multidimensionality of both reading comprehension in itself and the challenges imposed by reading test items. Both categories 4 and 6 include accounts that do not point directly to some aspect of comprehending the input text but rather to necessary knowledge of vocabulary or terminology used in the item. In Table 5 the distribution of teachers’ definitions over the whole set of 19 items is displayed. Items are also identified by official reading process category.

Table 5. Distribution of Raters’ Definitions of Item Difficulty.

Item	i1	i2	i3	i4	i5	i6	i7	i8	i9	i10	i11	i12	i13	i14	i15	i16	i17	i18	i19
Reading Process^a	3	4	1	1	2	4	3	3	3	1	3	2	2	4	4	2	2	3	4
R1	3	3	2	2	3	3	3	3	3	6	2	4	3	6	9	3	4	3	6
R2	3	3	4	2	3	9	3	9	9	4	9	4	3	6	2	3	4	4	6
R3	5	3	2	3	4	4	5	9	3	1	2	4	5	6	3	2	4	4	6
R4	2	3	2	2	2	2	2	2	2	2	2	2	2	6	2	2	4	2	6
R5	5	3	2	2	3	4	3	1	3	4	9	4	9	6	3	4	4	4	6
R6	5	4	9	2	2	4	1	4	1	1	3	4	3	6	4	3	4	3	6
R7	5	2	2	9	1	4	1	1	1	1	1	4	4	6	1	3	4	4	6
R8	5	1	1	1	9	4	5	1	1	1	1	9	9	6	1	1	4	4	4
R9	2	3	1	1	3	4	3	3	3	1	1	4	3	6	1	4	4	4	6
R10	2	3	2	2	3	2	3	1	2	2	3	2	3	6	1	2	4	4	6
R11	9	6	1	1	3	4	2	3	2	1	1	4	2	6	3	2	4	4	6

Comment: 9 = Missing data.

^aReading process according to test constructor.

Patterns in the table seem to support the assumption that for a couple of items, it is much easier to arrive at consensus about a description of the required type of comprehension than for other items. For instance, item no 14 reads:

i14. To which genre does [the title of the text] belong? Choose the correct alternative.

• ∘ Letter to the editor

• ∘ Column

• ∘ Blog post

• ∘ Report

To this item all raters provided explanations that referred to necessary knowledge about form and structure of text (category 6 in Table 4), as in the following examples:

R1: They need to recognize the text type, both in terms of layout and in terms of the way it is written.

R7: The student needs to understand what it is that defines the text type.

R10: To have knowledge about the form of different texts.

In contrast with many other items in the sample, i14 requires a limited amount of interpretive processing and little, if any, searching through the input text for information. Instead, the test taker should pay attention to global surface features of the text. Provided that s/he possesses adequate knowledge of genre, s/he should be able to identify the correct answer. R1 stresses the recognition of text type, while R10 highlights the need to “have knowledge.” Central to both of these accounts is familiarity with genre traits. All of the 11 raters agree that this is central for solving the item. In Table 5 we also find that many of the items for which teachers agree to a relatively large extent about item characteristics are items targeting reading process category 4. I15 may be an exception to this pattern, because the teachers’ descriptions of item difficulty are more varied. However, i15 is also a dubious example of category 4.

i15. Is [name of the author]’s attitude towards flying primarily positive or negative? Choose the correct alternative.

• ∘ Primarily positive

• ∘ Primarily negative

Justify your opinion with examples from the text.

_______________________________________________

_______________________________________________

_______________________________________________

Compared to other process 4 items, i15 does not focus on the evaluation of any textual features by using previous knowledge but rather on inferring an authorial perspective by integrating and interpreting different parts of the text. In the classification task none of the raters placed i15 in category 4, but rather in category 2 or 3 (Table 2), where the teachers’ item descriptions generally vary more substantially. I5 (category 2) may serve as an example:

i5: Explain why [name of main character] feels that the lesson is “pleasantly monotonous”.

_______________________________________________

_______________________________________________

_______________________________________________

When describing what the students need to understand to solve this task, the following accounts were given:

R2: That the boring lesson makes those who receive roses less “bubbling” so that [name of main character] will not become so exposed for the rapture in which he cannot participate.

R3: Be able to interpret that “pleasantly” means something positive and understand what he feels in the situation.

R4: “[name of teacher in the narrative] demonstrates equations that calm down even the most bubbling Valentine’s rapture.”

R6: That [name of main character] likes it when the lessons are monotonous.

R7: The student must understand the “figurative” description of the situation as well as direct descriptions.

The cited accounts represent various ways to conceive or explain textual challenge or difficulty in relation to the given item, and they all fall into different categories according to Table 4. Note that the issue here is not the extent to which these accounts provide more or less accurate understandings of the interaction between the input text and the item. What is important is the extent to which such differences in perceiving the comprehension and reading skills needed for a correct item response also reflect differences of a more profound content-related nature. It may, for instance, indicate variation in how teachers conceive task and text complexity in general, which may, in turn, affect both classroom instruction and assessment of reading comprehension.

From one perspective, variation in teachers’ description of item difficulty can, and perhaps should, be seen as a display of the multidimensionality inherent in many reading test items on an inferential level. As demonstrated in the example above, an item in which students are required to explain a character’s emotions will involve not only higher-order processing, such as integration of segments from different parts of the text, but also lower-order processing, such as interpretation of word meanings (in both the text and the item) and of sentences. From another perspective, validity of an achievement measure requires a certain degree of consensus among subject matter experts about what different items actually measure (Haladyna & Rodriguez, 2013). The fact that both categorization of items into reading process categories and open-ended descriptions of item difficulty vary considerably between teachers might indicate that the cognitive validity of the test is threatened. Implications of this will be discussed in the final section.

Discussion

The aim of the present study is to contribute relevant knowledge about validity in reading comprehension assessment by investigating (a) the extent to which specified reading process categories are conceptually conceivable to teachers and (b) the extent to which teachers agree on how to locate and define item difficulty in terms of expected text comprehension. Because test validity relies on adequate interpretation and use of test results and because teachers in Sweden both score the national tests and use test results formatively in the classroom after administration, their perception of test content is critical. Findings of the study indicate that there is a clear limit to the reliability with which reading test items are classified according to reading process category and the way in which item difficulty is located and defined by teachers.

However, while the reliability of classification was generally low, data suggested that some items were more consistently discerned than others. Agreement was, for instance, significantly higher for items dealing with examination of language and textual elements (category 4), both in terms of defining process level and in terms of perceiving item difficulty. Whether this can be explained by more easily discernible item characteristics or by a more intelligible instructional definition of the intended reading process cannot be determined from the existing data. However, it is worth noting that the instructional definition of category 4 is the only one containing an example of items included in the category (“Examples of this aspect of comprehension are to perceive that a given text is an article…,” Table 1). Definitions of the other three categories remain at a more abstract level. A concrete but simple implication of this might be that a more systematic description of the cognitive processes, including typical items and comprehension targets, could perhaps help to support a more reliable discernment of the sub-constructs.

The study also indicated higher interrater agreement for MC items than for CR items, although the difference was nonsignificant. It is possible that the MC format facilitates discernment of cognitive target because alternative answers (including the correct answer) are provided, thereby making the suggested route of textual understanding, or processing, more concrete. However, both of these suggestions (that agreement is higher when classifying examining content and language-items and when classifying MC items) need to be verified by future studies using a larger sample size.

On a more general note, it seems clear to assume that classification of items by reading process levels is a difficult task for teachers who are not trained as expert raters. This finding is consistent with some of the previous studies in the field (Alderson & Lukmani, 1989; DeStefano et al., 1997). Yet the complexity of the reading process, at the intersection of reader, text, and reading situation (Alderson, 2000; Snow, 2002), inevitably leaves substantial room for interpreting lexical, semantic, structural, and thematic constraints for comprehension differently. Therefore, the type of variation found in the present study is perhaps not very surprising. Following Alderson (2000),

[a]nswering a test question is likely to involve a variety of interrelated skills, rather than one skill only or even mainly. Even if there are separate skills in the reading process which one could identify by a rational process of analysis of one’s own reading behavior, it appears to be extremely difficult if not impossible to isolate them for the sake of testing or research (p. 49).

Thus, if test grades are to depend on subdivision of the construct, as is already the case in the Swedish national reading tests, clear evidence must be provided for both psychometric and psychological multidimensionality (cf. Henning, 1992). Empirical research has indicated that sub-constructs of reading are difficult to prove psychometrically (Meijer & van Gelderen, 2002; Spearritt, 1972; van Steensel et al., 2012). And while some studies have investigated the extent to which suggested sub-constructs can be made useful and conceivable to those who are responsible for development and scoring (Bachman et al., 1996; Lumley, 1993), few studies have examined the cognitive validity of such sub-constructs in testing systems where classroom teachers are responsible for the whole chain of scoring, interpreting, and using test results. Findings from the present study do not suggest that the concept of reading processes cannot be used for distinguishing sub-constructs of reading ability. Yet it demonstrates the necessity of collecting further empirical evidence for whether the postulated sub-constructs really exist as discernible psychological dimensions. Without such evidence the national test scores may be of limited use for reliable identification of students’ strengths and weaknesses.

The reported variance between teachers’ accounts of item difficulty and requested form of comprehension may certainly, following Alderson (2000), Cerdan et al. (2009), and Mosenthal (1996), refer to a multidimensionality of the comprehension and strategic thinking inherent in any successful solving of reading test items on inferential or reflection level. It has been pointed out that item difficulty as an empirical issue cannot necessarily be inferred from process level, response format, or even from the input text (Lumley et al., 2012; OECD, 2009; Rupp, Ferne, & Choi, 2006). Rather, item difficulty arises from a specific form of interaction between characteristics of certain segments in the text and the particular item. Yet by investigating how teachers locate and define the comprehension necessary for solving a set of given items, we also learn about patterns in the teachers’ perspectives. These perspectives are informative when it comes to understanding their perception of reading process levels, which in turn affects perhaps not their scoring but certainly their interpretation and use of student test scores.

According to Kane (2013), test validity should be concerned with the evidence that supports suggested interpretations and uses of test scores. In the present context, this implicates that the meaning of test results ought to be shared by those who score, interpret, and use them (i.e., qualified and experienced secondary teachers of Swedish). These teachers may be regarded as subject-matter experts (SMEs), in Haladyna’s and Rodriguez’ (2013) term, and should be able to distinguish the content and align the construct of the test to the goals in the curriculum. When SMEs disagree with the official categorization of specific items, the validity of those categorizations needs to be questioned. If the disagreement among SMEs is substantial, as in the present case, then the entire process of scale development and categorizing items needs to be evaluated thoroughly. An initial step for future test development might thus be to benchmark the process of scale development against comparable reading tests. If the tests are to include a subdivision of the construct into separate processes, or subscales, then strong emprical evidence for their existence must be collected.

The size of the data set used in the study obviously entails limitations to the type of conclusions that may validly be drawn from the results. As for rater reliability, the findings cannot be generalized to the population of teachers of Swedish. Yet if seen as an examination of the intelligibility and transparency of the test design itself, it should be expected that reliable classification does not appear on an aggregated level only but that it is present in any random sample of qualified and experienced teachers of Swedish. Furthermore, if treated as a case study, the findings reported may also provide some clues for future studies. For instance, if classification of subskills tested by reading test items is difficult, then it should be interesting to examine whether this is a general trait related to the broad definitions of the reading process categories themselves, or if it is related to other characteristics of the particular items examined here.

Conclusions

The findings reported in the study point to limitations of the cognitive validity primarily in the Swedish national reading test. Together with indications from some of the previous classification studies as well as from psychometric analyses of the dimensionality of reading, the findings also point to a potential weakness of the separation into cognitive processes itself. As long as sub-constructs are identified for the purpose of organizing the reading domain and ensuring a broad representation of the construct, then the consequences of limited classification reliability are perhaps minor. But when scores are reported on subscales or, as in the present case, when credits are required on items of several different subscales for test takers to earn a particular grade, then the consequences may be significant and stronger evidence for validity is also required. In line with Kane’s (2013) argument that what is to be validated is not tests or test scores but the proposed interpretations of test scores, it may be argued that regardless of whether the test has other qualities, such as a good representation of the construct at large or good predictive validity of future reading achievements, its capability of reliably identifying strengths and weaknesses on sub-construct level should be questioned.

As far as the content of the national test is expected to reflect curriculum goals of reading, it may also be inferred that scores on subscales open for a more detailed formative use of the test results. However, because teachers largely disagree on what separate items actually test as well as on what the primary difficulties of particular items are, such use of test scores, either in the classroom or elsewhere, does not appear to be justified. Reading process categories of the sort examined here (which are also found in the PIRLS and PISA tests) need therefore to be treated with caution. Until the divisibility of reading comprehension into process categories can be supported by substantial validity evidence, it should at least not form the basis of high-stakes dimensions of tests.

Acknowledgments

This research was partially supported by funding from Gunvor och Josef Anérs stiftelse.

Notes

¹ High-level questions, according to Rouet et al., refer to tasks that require “focus on a broader set of concepts or on concepts higher in a hierarchy” (i.e., inferential reading) (Rouet et al., p. 167). Low-level questions, in contrast, “focus on a single concept and require only superficial processing of the materials” (Ibid.).

² Ratings of item dimensionality in the Herman, Webb, & Zuniga (2007) study concerned whether items were unidimensional (targeting one topic) or multidimensional (targeting two topics). Depth-of-knowledge ratings concerned the degree of cognitive demand in a way similar to the ratings investigated in the present study.

³ The categories were Form an initial understanding; Develop an interpretation; Personally reflect and respond to the reading; and Demonstrate a critical stance (Pearson & Hamm, 2005).

⁴ The same test is used for students in both Swedish and Swedish as a second language. Thus, the test is used for measuring both L1 and L2 reading comprehension.

⁵ A slight moderation has been made in the third category, which in the Swedish framework includes reflection (see Table 1). In PIRLS the reflection aspect belongs to the fourth category.

⁶ In the study the teacher participants serve as “raters” and will interchangeably be referred to as teachers and raters.