Evaluating group differences in online reading comprehension: The impact of item properties

Abstract

This study examined group differences in online reading comprehension (ORC) using student data from the 2016 administration of the Progress in International Reading Literacy Study (ePIRLS). An explanatory item response modeling approach was used to explore the effects of item properties (i.e., item format, text complexity, and cognitive complexity), student characteristics (i.e., gender and language groups), and their interactions on dichotomous and polytomous item responses. The results showed that female students outperform male students in ORC tasks and that the achievement difference between female and male students appears to change text complexity increases. Similarly, the cognitive complexity of the items seems to play a significant role in explaining the gender gap in ORC performance. Students who never (or sometimes) speak the test language at home particularly struggled with answering ORC tasks. The achievement gap between students who always (or almost always) speak the test language at home and those who never (or sometimes) speak the test language at home was larger for constructed-response items and items with higher cognitive complexity. Overall, the findings suggest that item properties could help understand performance differences between gender and language groups in ORC assessments.

Keywords:

• Achievement gap
• ePIRLS
• explanatory item response modeling
• item properties
• online reading comprehension

Notes

1 In this study, country was not considered as a third-level clustering variable because of unbalanced and small sample sizes for the selected countries, based on the multilevel modeling guidelines in Maas and Hox (2005) study.

2 The interactions of both gender-item format and speaking the test language at home-text complexity were not statistically significant, and thus removed from the final version of Model 3.