Response Demands of Reading Comprehension Test Items: A Review of Item Difficulty Modeling Studies

References

• Alderson, J. C., de Jong, J., Kirsch, I., Lafontaine, D., Lumley, T., Mendelovits, J., & Searle, D. (2009). How can we predict the difficulty of PISA reading items? The process of describing item difficulty. Paper presented in the Language Testing Forum, University of Bedfordshire, England.
   Google Scholar
• Baker, M. (2016, May 26). Is there a reproducibility crisis? Nature, 533, 452–454. doi:10.1038/533452a
   PubMed Web of Science ®Google Scholar
• Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1984). Classification and regression trees. Wadsworth.
   Google Scholar
• Drum, P. A., Calfee, R. C., & Cook, L. K. (1981). The effects of surface structure variables on performance in reading comprehension tests. Reading Research Quarterly, 16(4), 486–514. doi:10.2307/747313
   Web of Science ®Google Scholar
• Ferrara, S., & Qunbar, S. ((2022 in press)). Validity arguments for AI-based automated scores: Essay scoring as an illustration. In H. Michaels (Ed.), Journal of Educational Measurement.
   Google Scholar
• Ferrara, S., Svetina, D., Skucha, S., & Murphy, A. (2011). Test design with performance standards and achievement growth in mind. Educational Measurement: Issues and Practice, 30(4), 3–15. doi:10.1111/j.1745-3992.2011.00218.x
   Google Scholar
• Ferrara, S. (2017). Aligning item response demands with knowledge and skill requirements in achievement level descriptors: An approach to achieving full alignment and engineering cut scores. In D. Lewis(Chair), Engineered cut scores: Aligning standard setting methodology with contemporary assessment design prnciples. Coordinated session in the annual meeting of the National Council on Measurement in Education, San Antonio, TX.
   Google Scholar
• Ferrara, S., Steedle, J., & Frantz, R. (2018). Item design with test score interpretation in mind. In J. Steedle (Ed.), Item difficulty modeling: Lessons learned and future directions. New York: Coordinated session at the annual meeting of the National Council on Measurement in Education.
   Google Scholar
• Ferrara, S., & Qunbar, S. (2021, June 22). Validity arguments for automated scoring engines, with illustrations for writing and science. In H. Michaels (Ed.), The importance of validity arguments for innovative AI-enhanced online assessments.(Symposium). National Conference on Student Assessment (virtual).
   Google Scholar
• Fischer, G. H. (1995). The linear logistic test model. In G. H. Fischer & I. W. Molenaar (Eds.), Rasch models. Springer. doi:10.1007/978-1-4612-4230-7_8
   Google Scholar
• Freedle, R., & Kostin, I. (1992). The prediction of SAT reading comprehension item difficulty for expository prose passages. ETS Research Report RR-91-29. Retrieved from https://onlinelibrary.wiley.com/doi/abs/10.1002/j.2333-8504.1991.tb01396.x
   Google Scholar
• Gorin, J. S., & Svetina, D. (2008). SAT critical reading Q-matrix study: LLTM analysis of Q-matrix attributes. Report submitted to the College Board. Available from the authors.
   Google Scholar
• Guttman, L., & Schlesinger, I. M. (1966). Development of diagnostic analytical and mechanical ability tests through facet design and analysis. Report no. CRP-OE-4-21-014. Jerusalem, Israel: Israel Institute of Applied Social Research. Retrieved from https://files.eric.ed.gov/fulltext/ED010590.pdf
   Google Scholar
• Kirsch, I. S., & Mosenthal, P. B. (1988). Understanding document literacy: Variables underlying the performance of young adults, ETS Research Report No. RR-88-62. Princeton, N J: Educational Testing Service.
   Google Scholar
• Kirsch, I. S., & Mosenthal, P. B. (1990). Exploring document literacy: Variables underlying the performance of young adults. Reading Research Quarterly, 25(1), 5–30. doi:10.2307/747985
   Web of Science ®Google Scholar
• Kulesz, P. A., Francis, D. J., Barnes, M. A., & Fletcher, J. M. (2016). The influence of properties of the test and their interactions with reader characteristics on reading comprehension: An exploratory item response study. Journal of Educational Psychology, 108(8), 1078–1097. doi:10.1037/edu0000126
   Google Scholar
• Luecht, R. M. (2013). Assessment engineering task model maps, task models and templates as a new way to develop and implement test specification. Journal of Applied Testing Technology, 14. Retrieved from https://www.testpublishers.org/assets/documents/test%20specifications%20jatt%20special%20issue%2013.pdf
   Google Scholar
• Luecht, R., & Burke, M. (2020). Reconceptualizing items: From clones and automatic item generation to task model families. In R. Lissitz & H. Jiao (Eds.), Applications of artificial intelligence to assessment. Information Age Publishers.
   Google Scholar
• Lumley, T., Routitsky, A., Mendelovits, J., & Ramalingam, D. (2012, April). A framework for predicting item difficulty in reading tests. Paper presented at the annual meeting of the American Educational Research Association, Vancouver, BC. Retrieved from https://research.acer.edu.au/cgi/viewcontent.cgi?article=1004&context=pisa
   Google Scholar
• Mosenthal, P. B. (1996). Understanding the strategies of document literacy and their conditions of use. Journal of Educational Psychology, 88(2), 314–332. doi:10.1037/0022-0663.88.2.314
   Web of Science ®Google Scholar
• Mosenthal, P. B. (1998). Defining prose task characteristics for use in computer-adaptive testing and instruction. American Educational Research Journal, 35(2), 269–307. doi:10.3102/00028312035002269
   Web of Science ®Google Scholar
• Randall, J. (2021). Commentary: “Color-neutral” is not a thing: Redefining construct definition and representation through a justice-oriented lens. Educational Measurement: Issues and Practice, 40(4), 82–90. doi:10.1111/emip.12429
   Web of Science ®Google Scholar
• Rowe, M., Ozuru, Y., & McNamara, D. (2006). An analysis of standardized reading ability tests: What do questions actually measure? In ICLS 2006 - International Conference of the Learning Sciences, Proceedings (vol. 2, pp. 627–633). Retrieved from https://repository.isls.org/handle/1/3566
   Google Scholar
• Sano, M. (2016, April). Improvements in automated capturing of psycho-linguistic features in reading assessment text. Paper presented at the annual meeting of the National Council on Measurement in Education, Washington, DC.
   Google Scholar
• Scheuneman, J., Gerritz, K., & Embretson, S. (1991). Effects of prose complexity on achievement test item difficulty. ETS Research Report ETS-RR-91-43. Princeton, NJ: Educational Testing Service. Retrieved from https://files.eric.ed.gov/fulltext/ED389717.pdf
   Google Scholar
• Schneider, M. C., Chen, J., & Nichols, P. (2021). Using principled assessment design and IDM to connect hybrid adaptive instructional and assessment systems: Proof of concept. Proceedings from the Human Computer Interaction International 2021 meeting. Springer.
   Google Scholar
• Sheehan, K., & Mislevy, R. J. (1990). Integrating cognitive and psychometric models to measure document literacy. Journal of Educational Measurement, 27(3), 255–272. doi:10.1111/j.1745-3984.1990.tb00747.x
   Web of Science ®Google Scholar
• Sheehan, K. M. (2016). A review of evidence presented in support of three claims in the validity argument for the TextEvaluator text analysis took. Research report ETS RR-16-12. Retrieved from https://www.semanticscholar.org/paper/A-Review-of-Evidence-Presented-in-Support-of-Three-Sheehan/076063c9d0a9463e4c5c03b09adf0663344f2c3e
   Google Scholar
• Tinkelman, S. (1947). Difficulty prediction of test items. Teachers College Contributions to Education, 941, 55.
   Google Scholar
• Toyama, Y. (2019). What makes reading difficult? An investigation of the contribution of passage, task, and reader characteristics on item difficulty, using explanatory item response models. (Doctoral dissertation). University of California, Berkeley.
   Google Scholar
• Valencia, S. W., Wixson, K. K., Ackerman, T., & Sanders, E. (2017). Identifying text-task-reader interactions related to item and block difficulty in the national assessment of educational progress reading assessment. Study commissioned by the NAEP Validity Studies Panel. Retrieved from https://www.air.org/sites/default/files/downloads/report/Identifying-Text-Task-Reader-Interactions-Related-to-Item-and-Block-Difficulty-NAEP-Oct-2017.pdf
   Google Scholar
• Webb, N. L. (1997). Criteria for alignment of expectations and assessments in mathematics and science education. Research monograph no. 6. CCSSO. Retrieved from https://eric.ed.gov/?id=ED414305
   Google Scholar