Addressing omitted prior achievement bias in international assessments: an applied example using PIRLS-NPD matched data

Abstract

Omitted prior achievement bias is pervasive in international assessment studies and precludes causal inference. For example, reported negative associations between student-oriented teaching strategies and student performance are against expectations and might actually reflect omitted prior achievement bias. Namely, that these teaching strategies are negatively correlated with unobserved prior achievement performance, because teachers offer more support to lower performing students, and not that these strategies cause lower performance. This paper examines omitted prior achievement bias in teaching effects with prior achievement data available for students in England participating in the Progress in International Reading Literacy Study. Further, it proposes an analytical approach to account for omitted prior achievement bias in international estimates. The paper argues that the bias is not simply a technical artefact, but reflects educational mechanisms unobserved in international assessment studies, which can be captured with matched assessment data-sets or with evidence from previous studies. Estimates of these mechanisms can be used to postulate scenarios of the bias across education systems and thereby adjust international estimates of teaching effects as if prior achievement were observed. Potentials and limitations of this approach for studying educational effectiveness with international assessment data are discussed.

Keywords:

• Omitted prior achievement bias
• international large-scale assessments
• teaching strategies
• reading performance
• PIRLS
• National Pupil Database

Acknowledgements

The authors are grateful to participants of the EARLI SIG Educational Effectiveness 2016 and two anonymous reviewers for their helpful comments. Dr Daniel Caro was granted access to the NPD-PIRLS matched data-set by the Department for Education (DfE), UK through OUCEA.