League tables in educational evidence-based policy-making: can we stop the horse race, please?

ABSTRACT

International large-scale student assessments (ILSAs) in education represent a valuable source of information for policy-makers, not only on student achievements, but also on their relationship with different contextual factors. The results are partly described in the official studies’ reports; more can be derived from the publicly released data sets. However, league tables are often the only evidence used in policy debates and decisions on education. Indeed, the comparison of student achievement across the participating educational systems is a legitimate proxy for estimating countries’ development and productivity, but the use of league tables more often turns into ‘horse-ranking’, ignoring the contexts of teaching and learning. This is often supported by the media, turning the use of results into their abuse. The purpose of this paper is to discuss the use and misuse of league tables in reporting ILSA results, vs. the use of data for in-depth analysis in order to make informed decisions.

KEYWORDS:

• Student achievement
• league tables
• secondary analysis
• decision-making
• policy-making

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes on contributors

Dr Eva Klemenčič is a researcher, head of the Center for Applied Epistemology at the Educational Research Institute (Slovenia), which conducts all OECD and IEA international large-scale student assessments in which Slovenia participates. She is one of the founders of the international research network that conducts research on the use of the international large-scale student assessments datasets (ILSA) to the national policy-making in the field of education. Her main research areas are: ILSA, citizenship education, textbook analysis, didactics of sociology, sociology of education, and theories of knowledge.

Plamen Vladkov Mirazchiyski holds a master and PhD degrees in Social Pedagogy and has a Bachelor's degree in Pre-school and Primary Education. Since 2008 Plamen has been employed at the International Association for the Evaluation of Educational Achievement in Hamburg (Germany), where he was the Deputy Head of the Research and Analysis Unit for three years. Since July 2016, he has become the Chief Executive Officer the International Educational Research and Evaluation Institute (INERI). His main research interests are reading literacy, mathematics and science education, civic and citizenship education, ICT in education, methodology of large-scale assessments and psychometrics.

ORCID

Eva Klemenčič http://orcid.org/0000-0002-3674-9406

Plamen Vladkov Mirazchiyski http://orcid.org/0000-0003-3452-6046

Notes

1. In this article, by the term ‘context’ we understand the political, socioeconomic, cultural, economic and other possible contexts where the teaching and learning take place. For example, the IEA International Civic and Citizenship Education Study’s contextual framework distinguishes the following levels: (1) context of the wider community (factors that can be found at local, regional and national levels, in some countries supra-national level might be relevant as well); (2) context of schools and classrooms (factors related to the instruction students receive, the school culture and the general school environment); (3) context of home environment (home background and the social out-of-school environment of the student), and (4) context of the individual (individual characteristics of the student) (Schulz et al. 2008). Another important distinction could be made by grouping contextual factors according to those related to either antecedents or processes (Schulz et al. 2008). In other ILSAs, the contexts are defined in a similar fashion. All ILSAs collect data on these different contexts. In the current article, we distinguish the contextual variables in the data sets from the contexts where teaching and learning, unless specified otherwise.

2. OECD assessments differ significantly from IEA assessments in two ways: (1) IEA studies are grade based whereas OECD’s PISA targets a specific age group and (2) OECD PISA does not consider curricula as a base, and thus the value of the results is criticized as lacking relevance for educational policy-making (Klemencic 2010; Wagemaker 2014).

3. This, of course, does not mean that school characteristics do not matter.

4. Policy borrowing and policy learning differ. The policy learning approach offers a more promising way to use international experience to inform policy-making:

Policy borrowing involves searching the international experience for transferable ‘best practice’; policy learning uses this experience for a wider range of purposes, including understanding one’s own system better, identifying common trends and pressures that affect all systems, clarifying alternative policy strategies and identifying issues raised by each strategy (Raffe 2011, 2).