Engineering a Twenty-First Century Reading Comprehension Assessment System Utilizing Scenario-Based Assessment Techniques

Abstract

The construct of reading comprehension has changed significantly in the twenty-first century; however, some test designs have not evolved sufficiently to capture these changes. Specifically, the nature of literacy sources and skills required has changed (wrought primarily by widespread use of digital technologies). Modern theories of comprehension and discourse processes have been developed to accommodate these changes, and the learning sciences have followed suit. These influences have significant implications for how we think about the development of comprehension proficiency across grades. In this paper, we describe a theoretically driven, developmentally sensitive assessment system based on a scenario-based assessment paradigm, and present evidence for its feasibility and psychometric soundness.

Keywords:

• reading comprehension assessment
• reading for understanding
• scenario-based assessment

ACKNOWLEDGEMENTS

We are grateful to the Institute of Education Sciences and Educational Testing Service for sponsoring and supporting this research. We would like to also like to thank Michael Kane and Robert Mislevy for their intellectual insights and thoughtful comments; Andre Rupp, Paul Deane and Heather Buzick for their review comments; Kelsey Dreier and Kim Fryer for their editorial assistance. We also wish to express our gratitude to the all of our RfU and Cognitively Based Assessment as, of, & for, Learning (CBAL^TM) colleagues, who have been and are contributing to this ongoing enterprise.

Notes

1 The CBAL initiative and Institute for Education Sciences grant had overlapping aims. It is beyond the scope of this article to review relevant CBAL publications. The interested reader is referred to the ETS website (https://www.ets.org/research/topics/) for a more complete bibliography.

2 Linguistic complexity is used here to refer to the linguistic demands such as text cohesion, syntactic complexity, and the level of vocabulary sophistication that may impact a reader’s ability to from a coherent model of the text (see McNamara, Graesser, & Louwerse, 2012). Linguistic complexity may contribute to item difficulty independent of the task and content demands.

3 Reading comprehension is not directly observable, but the quality of a student’s mental model can be inferred from the evidence gathered through a sample items and tasks.

Additional information

Funding

The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305F100005 to the Educational Testing Service as part of the Reading for Understanding Research (RFU) Initiative. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education.