Validation of sub-constructs in reading comprehension tests using teachers’ classification of cognitive targets

References

• Alderson, J. C. (2000). Assessing reading. Cambridge: Cambridge University Press.
   Google Scholar
• Alderson, J. C., & Lukmani, Y. (1989). Cognition and reading: Cognitive levels as embodied in test questions. Reading in a Foreign Language, 5(2), 253–270.
   Google Scholar
• Bachman, L. F., Davidson, F., & Milanovic, M. (1996). The use of test method characteristics in the content analysis and design of EFL proficiency tests. Language Testing, 13(2), 125‒150. doi:10.1177/026553229601300201
   Google Scholar
• Best, R., Ozuru, Y., Floyd., R., & McNamara, D. S. (2006). Children’s text comprehension. Effects of genre, knowledge, and text cohesion. In S. A. Barab, K. E. Hay, & D. T. Hickey (Eds.), Proceedings of the seventh international conference of the learning sciences (pp. 37–42). Mahwah, NJ: Erlbaum.
   Google Scholar
• Brutten, S. R., Perkins, K., & Upshur, J. A. (1991). Measuring growth in ESL reading. Paper presented at the 13th Annual Language Testing Research Colloquium. Princeton, NJ.
   Google Scholar
• Campbell, J. R. (2005). Single instrument, multiple measures: Considering the use of multiple item formats to assess reading comprehension. In S. G. Paris & S. A. Stahl (Eds.), Children’s reading comprehension and assessment (pp. 347–368). Mahwah, NJ: Lawrence Erlbaum Ass.
   Google Scholar
• Cerdan, R., Vidal-Abarca, E., Martinez, T., Gilabert, R., & Gil, L. (2009). Impact of question-answering tasks on search processes and reading comprehension. Learning and Instruction, 19(1), 13–27. doi:10.1016/j.learninstruc.2007.12.003
   Web of Science ®Google Scholar
• Davis, F. B. (1968). Research in comprehension in reading. Reading Research Quarterly, 3, 499–545. doi:10.2307/747153
   Web of Science ®Google Scholar
• DeStefano, L., Pearson, P., & Afflerbach, P. (1997). Content validation of the 1994 NAEP in Reading: Assessing the relationship between the 1994 assessment and the reading framework. In R. Linn, R. Glaser, & G. Bohrnstedt (Eds.), Assessment in transition. 1994 trial state assessment report on reading: Background studies (pp. 1–50). Stanford, CA: The National Academy of Education.
   Google Scholar
• EACEA; Eurydice. (2009). National testing of pupils in Europe: Objectives, organisation and use of results. Brussels, Belgium: Eurydice.
   Google Scholar
• Field, J. (2013). Cognitive validity. In A. Geranpayeh & L. Taylor (Eds), Examining listening: Research and practice in assessing second language (pp. 77‒151). Cambridge: Cambridge University Press.
   Google Scholar
• Francis, D. J., Snow, C. E., August, Carlson, C. D., Miller, J., & Iglesias, A. (2006). Measures of reading comprehension: A latent variable analysis of the diagnostic assessment of reading comprehension. Scientific Studies of Reading, 10(3), 301–322. doi:10.1207/s1532799xssr1003_6
   Web of Science ®Google Scholar
• Glaser, R., & Linn, R. (1994). Assessing the content of the trial state assessment of the National Assessment of Educational Progress in reading. Educational Assessment, 2(3), 273–274. doi:10.1207/s15326977ea0203_4
   Google Scholar
• Gwet, K. L. (2008). Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology, 61, 29–48. doi:10.1348/000711006X126600
   PubMed Web of Science ®Google Scholar
• Haladyna, T. M., & Rodriguez, M. C. (2013). Developing and validating test items. New York, NY: Routledge.
   Google Scholar
• Hallgren, K. A. (2012). Computing inter-rater reliability for observational data: An overview and tutorial. Tutorials in quantitative methods for Psychology, 8(1), 23–34. doi:10.20982/tqmp.08.1.p023
   PubMedGoogle Scholar
• Henning, G. (1992). Dimensionality and construct validity of language tests. Language Testing, 9(1), 1‒11. doi:10.1177/026553229200900102
   Google Scholar
• Herman, J. L., Webb, N. M., & Zuniga, S. A. (2007). Measurement issues in the alignment of standards and assessments: A case study. Applied Measurement in Education, 20(1), 101–126.
   Google Scholar
• IEA. (2009). PIRLS 2011 Assessment framework. Chestnut Hill, MA: Boston College.
   Google Scholar
• Kane, M. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50, 1–73. doi:10.1111/jedm.2013.50.issue-1
   Web of Science ®Google Scholar
• Keenan, J. M., Betjemann, R. S., & Olson, R. K. (2008). Reading comprehension tests vary in the skills they assess: Differential dependence on decoding and oral comprehension. Scientific Studies of Reading, 12(3), 281–300. doi:10.1080/10888430802132279
   Web of Science ®Google Scholar
• Khalifa, H., & Weir, C. (2009). Examining reading: Research and practice in assessing second language learning. New York, NY: Cambridge University Press.
   Google Scholar
• Koretz, D. (2008). Measuring up: What educational testing really tells us. Cambridge, MA: Harvard University Press.
   Google Scholar
• Krippendorff, K. (1980). Content analysis: An introduction to its methodology. Beverly Hills, CA: Sage Publications.
   Google Scholar
• Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159–174. doi:10.2307/2529310
   PubMed Web of Science ®Google Scholar
• Langer, J. (1995). Envisioning literature: Literary understanding and literature instruction. New York, London: Teachers College Press.
   Google Scholar
• Lombardi, A., Seburn, M., Conley, D., & Snow, E. (2010). A generalizability investigation of cognitive demand and rigor ratings of items and standards in an alignment study. Paper presented at the annual conference of the American Educational Research Association, Denver, CO, April 2010. Available at: https://files.eric.ed.gov/fulltext/ED509419.pdf
   Google Scholar
• Lumley, T. (1993). The notion of subskills in reading comprehension tests: An EAP example. Language Testing, 10(3), 211‒234. doi:10.1177/026553229301000302
   Google Scholar
• Lumley, T., Routitsky, A., Mendelovits, J., & Ramalingam, D. (2012). A framework for predicting item difficulty in reading tests. Paper presented at the American Educational Research Association Meeting, Vancouver, BC.
   Google Scholar
• McNamara, D. S., & Magliano, J. (2009). Toward a comprehensive model of comprehension. Psychology of Learning and Motivation, 51, 297–384.
   Web of Science ®Google Scholar
• Meijer, J., & van Gelderen, A. (2002). Lezen voor het leven: Een empirische vergelijking van een nationale en een internationale leesvaardigheidspeiling. Amsterdam, Netherlands: SCO-Kohnstamminstituut.
   Google Scholar
• Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13–103). New York, NY: American Council on Education.
   Google Scholar
• Mosenthal, P. B. (1996). Understanding the strategies of document literacy and their conditions of use. Journal of Educational Psychology, 88(2), 314–332. doi:10.1037/0022-0663.88.2.314
   Web of Science ®Google Scholar
• OECD. (1999). PISA 2000. Measuring student knowledge and skills. A new framework for assessment. Retrieved from www.oecd.org/
   Google Scholar
• OECD (2009). PISA 2009. Assessment framework: Key competencies in reading, mathematics and science. Retrieved from www.oecd.org/.
   Google Scholar
• Pearson, P. D., & Hamm, D. N. (2005). The assessment of reading comprehension: A review of practices: Past, present, and future. In S. G. Paris & S. A. Stahl (Eds.), Children’s reading comprehension and assessment (pp. 13–70). Mahwah, NJ: Law. Erlbaum Ass.
   Google Scholar
• Rost, D. H. (1993). Assessing different components of reading comprehension: Fact or fiction? Language Testing, 10(1), 79–92. doi:10.1177/026553229301000105
   Google Scholar
• Rouet, J.-F., Vidal-Abarca, E., Erboul, A. B., & Millogo, V. (2001). Effects of information search tasks on the comprehension of instructional text. Discourse Processes, 31(2), 163–186. doi:10.1207/S15326950DP3102_03
   Web of Science ®Google Scholar
• Rupp, A., Ferne, T., & Choi, H. (2006). How assessing reading comprehension with multiple-choice questions shapes the construct: A cognitive processing perspective. Language Testing, 23(4), 441–474. doi:10.1191/0265532206lt337oa
   Google Scholar
• Rupp, A. A. (2012). Psychological vs. psychometric dimensionality in reading assessment. In J. Sabatini, E. R. Albro, & T. O’Reilly (Eds.), Measuring up: Advances in how we assess reading ability (pp. 135–152). New York, NY: Rowan & Littlefield Education.
   Google Scholar
• Schedl, M., Gordon, A., Carey, P. A., & Tang, K. L. (1996). An analysis of the dimensionality of TOEFL reading comprehension items. Princeton, NJ: Educational Testing Service.
   Google Scholar
• Shohamy, E., Donitsa-Schmidt, S., & Ferman, I. (1996). Test impact revisited: Washback effect over time. Language Testing, 13(3), 298–317. doi:10.1177/026553229601300305
   Google Scholar
• Snow, C. E. (Ed.). (2002). Reading for understanding: Toward an R&D program in reading comprehension. Santa Monica, CA: Rand.
   Google Scholar
• Solheim, O. J., & Skaftun, A. (2009). The problem of semantic openness and constructed response. Assessment in Education: Principles, Policy & Practice, 16(2), 149–164. doi:10.1080/09695940903075909
   Google Scholar
• Song, M.-Y. (2008). Do divisible subskills exist in second language (L2) comprehension? A structural equation modeling approach. Language Testing, 25(4), 435–464. doi:10.1177/0265532208094272
   Web of Science ®Google Scholar
• Spearritt, D. (1972). Identification of sub-skills of reading comprehension by maximum likelihood factor analysis. Reading Research Quarterly, 8, 92–111. doi:10.2307/746983
   Web of Science ®Google Scholar
• Stobart, G. (2008). Testing times: The uses and abises of assessment. London, UK: Routledge.
   Google Scholar
• Swedish National Agency for Education. (2011). Curriculum for the compulsory school, preschool class and the leisure-time centre 2011. Stockholm, Swedish National Agency for Education.
   Google Scholar
• Swedish National Agency for Education. (2014). Teacher guideline. National test in Swedish/Swedish as a second language 2014. Part A. To read and to comprehend. Stockholm: Swedish National Agency for Education.
   Google Scholar
• Tengberg, M. (2017). National reading tests in Denmark, Norway, and Sweden: A comparison of construct definitions, cognitive targets, and response formats. Language Testing, 34(1), 83–100. doi:10.1177/0265532215609392
   Web of Science ®Google Scholar
• van den Bergh, H. (1990). On the construct validity of multiple-choice items for reading comprehension. Applied Psychological Measurement, 14(1), 1–12. doi:10.1177/014662169001400101
   Web of Science ®Google Scholar
• van den Broek, P. (2012). Individual and developmental differences in reading comprehension: Assessing cognitive processes and outcomes. In J. P. Sabatini, E. R. Albro, & T. O’Reilly (Eds.), Measuring up: Advances in how to assess reading ability (pp. 39–58). New York, NY: Rowman & Littlefield Education.
   Google Scholar
• van Steensel, R., Oostdam, R., & van Gelderen, A. (2012). Assessing reading comprehension in adolescent low achievers: Subskills identification and task specificity. Language Testing, 30(1), 3–21. doi:10.1177/0265532212440950
   Web of Science ®Google Scholar