Performance status and change – measuring education system effectiveness with data from PISA 2000–2009

Abstract

Reports of international large-scale assessments tend to evaluate and compare education system performance based on absolute scores. And policymakers refer to high-performing and economically prosperous education systems to enhance their own systemic features. But socioeconomic differences between systems compromise the plausibility of those comparisons and references. The paper applies conceptual and methodological approaches from educational effectiveness research to investigate how effectively education systems perform and how effectively they change their performance over time by accounting for socioeconomic differences between systems and cohorts (assessment cycles). Data from 4 cycles of the Programme for International Student Assessment (PISA) are analysed. Results indicate that the quality of systems is evaluated differently if assessed by absolute performance scores or effectiveness measures. The study contributes to methodological developments of effectiveness research in international large-scale assessments and provides relevant information for policymakers to further look into policies, structures, and reform measures that have favoured effectiveness.

Keywords:

• education system effectiveness
• PISA
• performance trends
• educational policymaking

Notes on contributors

Jenny Lenkeit is a post-doctoral researcher at the Amsterdam Centre for Inequality Studies and the Research Institute of Child Development and Education at the University of Amsterdam. She received her PhD in Education from the University of Hamburg and a Master's degree in Empirical Educational Research and Comparative Education from the Humboldt University of Berlin. Her research interests are focused on methodological approaches in effectiveness research, cross-cultural comparisons, impact of international large-scale assessments on educational policy, and ethnicity-related inequalities in education.

Daniel H. Caro is a Research Fellow at the Oxford University Centre for Educational Assessment (OUCEA), Department of Education, University of Oxford. He completed a PhD in Education at the Freie Universität Berlin and a Master's degree in Interdisciplinary Studies at the University of New Brunswick. He is alumnus of the International Max Planck Research School on the Life Course (LIFE). His research interests include education inequality, international large-scale student assessments, quality of examination marking, mixed models in cross-sectional and longitudinal settings, and causal inference with observational data.

Notes

1. Five phases of school and system improvement are outlined by the authors: (1) understanding the organisational culture of the school; (2) action research and individual initiative; (3) manage change and the emphasis on leadership; (4) building capacity for learning at the local level (5) towards systemic improvement (Hopkins et al., 2011).

2. The deviations from the unconditional sample mean can only be approximations to the average performance scores in PISA 2009 minus the overall performance mean. This is first because multilevel modelling adjusts for precision in the calculation of overall means and random effects. Second, we use a subsample of education systems whose average performance differs from that of all in PISA 2009 participating systems. Third, we had to draw subsamples of each education system and cohort for computing reasons.

3. See also video footage in the context of the Lessons from PISA for the United States, Strong Performers and Successful Reformers in Education (OECD, 2011), available at http://www.pearsonfoundation.org/oecd/