Exploration of direct and indirect associations of system-level policy-amenable variables with reading literacy performance

Abstract

As expectations of the economic impact of educational attainment are soaring (Hanushek & Woessmann, 2009) and conjectures about successful national educational reforms (Mourshed, Chijioke, & Barber, 2010) are welcomed by educational policy-makers in many countries, a careful assessment of the empirical evidence for these kinds of claims is needed. In this article, we present a methodology that was applied to an international data set. A multi-level model of education was used to present a hypothetical scenario, indicated as the “implementation scenario”. The scenario was tested on the Programme for International Student Assessment (PISA) 2009 data set by means of multi-level structural equation modelling. Although we find some evidence for direct effects and some support for straightforward implementation, the overall impact of malleable conditions at the system and school level appears disappointingly small. A theoretical strand of literature that would account for “limited malleability” is referred to in discussing these results.

Keywords:

• educational effectiveness
• policy scenario
• implementation
• multi-level structural equation modelling
• accountability
• school autonomy
• choice

Notes on contributors

Jaap Scheerens is professor emeritus at the University of Twente in The Netherlands, and visiting professor at the University of Bristol (UK). He has published 20 books and about 100 articles in scientific journals, mainly addressing educational effectiveness and educational evaluation.

Hans Luyten ([email protected]) is an Associate Professor at the Faculty of Behavioural, Management and Social Sciences (Department of Research Methodology, Measurement and Data Analysis) of the University of Twente in The Netherlands and a Honorary Professor at the CEM Centre for Evaluation and Monitoring at Durham University in the UK. His research interests include educational effectiveness, social inequality, educational policy, and international comparisons.

Stéphanie M. van den Berg is associate professor at the University of Twente, where she develops Bayesian psychometric methods for applications in educational and genetics research. Her main interest is in the interplay of nature and nurture in the expression of psychological traits and educational achievement.

Cees A. W. Glas is the chair of the Department of Research Methodology, Measurement and Data Analysis, at the Faculty of Behavioural, Management and Social Sciences of the University of Twente, Enschede, The Netherlands. The focus of his scientific work is the development and application of statistical models in educational and psychological testing and in large-scale surveys.

Notes

1. These data were obtained from panels of educational experts and stakeholders in the participating country, and submitted to elaborate cross-checking at national and international levels.

2. When we use the term “effects” in this context, these should be interpreted as “statistical effects”; causal interpretation of these “effects” will be taken up further on in the text.

3. In the sense that low stratification at the system level was associated with high stratification/selectivity at the school level.

4. In the sense that high stratification at the system level was associated with high stratification/selectivity at the school level.