Standardisation and diversity in international assessments: barking up the wrong tree?

ABSTRACT

This article organises potential areas of criticism or challenges embedded in the design and administration of standardised assessments of learning levels in order to promote dialogue and research on educational assessments. The article begins by addressing debates around epistemological claims: issues that pertain to testing in general and issues that are particular to standardised testing. Then, it addresses some political attributes of international tests so as to situate the debates beyond feasibility, attributes and scope-related issues. The article claims that the field of education testing has identified a number of issues and challenges stemming from diversity, and has developed methods and procedures to address many of them. From this viewpoint, testing is just like any other domain of scientific enquiry. However, international assessments of learning outcomes are not necessarily, or primarily, scientific endeavours; they are political devices and therefore should be scrutinised considering scientific attributes as well as some political features that, even if intertwined with technicalities, go well beyond them. Thus, critiques of international assessments would be better framed if their political attributes are taken as organising principles of the criticism, alongside those elements that pertain to their technical attributes, since these are not incidental but deeply interlinked.

KEYWORDS:

• Cultural and linguistic diversity
• educational testing
• learning outcomes
• politics of testing
• standardised assessments

Acknowledgements

Over the past years, several colleagues have contributed through their publications and professional exchanges with the author, and on most occasions in a rather inadvertent manner, to the development of the ideas presented in this essay. The author is particularly grateful to Salim Abu-Rabia, Manuel Cardoso, Luis Crouch, Santiago Cueto, Steve Dept, Ronald K Hambleton, Mary Hamilton, Bryan Maddox, Liliana Miranda, Ralf St. Clair, John Strucker, Juan Fernando Vega and Bruno Zumbo. Paola Castro provided comments on the final revision of the text. The debates at the seminar ‘The challenges of diversity’ organised by the Laboratory of International Assessment Studies (July 2015) triggered and informed the initial ideas and motivation for writing this article. The anonymous reviewing process has improved this text in a substantial manner. None of the above-mentioned individuals is responsible for any shortcomings or mistakes, which are entirely an author’s responsibility.

This article has been written as part of the author’s research responsibilities at Universidad del Pacífico (Peru). There has been no specific funding for it beyond the notional costs corresponding to the allocation of time and the use of Universidad del Pacífico’s equipment and premises.

Disclosure statement

No potential conflict of interest was reported by the author.

Notes

1. The concrete is concrete because it is the concentration of many determinations, hence unity of the manifold (translated by the author).

2. This is the approach followed by the International Association for the Evaluation of Educational Achievement (IEA) since its inception in the late 1950s. See, for instance, Mullis, Martin, Minnich, Drucker, & Ragan Eds. (2012), Mullis, I. V. S., Martin, M., Minnich, C., Stanco, G., Arora, A., Centurino, V., & Castle, C. Eds. (2012), and Mullis and Martin (2012).

3. This is the approach followed by the OECD in its Programme for International Student Assessment (PISA) (OECD, 2013a) and its Programme for the International Assessment of Adult Competencies (PIAAC) (OECD, 2013b), and originally devised for the International Adult Literacy Survey (IALS) (Murray, Kirsh, & Jenkins, 1997; St. Clair, 2012) in the 1990s.

4. Study originally conceived as a sequel to IALS (OECD & Canada: Statistics Canada, 2005).

5. The stimulus, question (question 3) and scoring rubric are all publically available at https://nces.ed.gov/surveys/all/Items.asp?sub=yes&SectionID=2&CatID=3.

6. It is interesting to note that even the most stringent critics of positivism talk about ‘data collection’ in their own research activities, as if data were wild berries that could actually be collected. Data, just like narratives, are produced using concepts, theoretical frameworks and from particular standpoints.

7. As an illustration pertaining to a much less complex issue, we can mention that PISA is conducted in some countries/territories (Iceland, Liechtenstein, Luxembourg, Macao-China and Qatar for the 2012 round) on a census basis because of their population size; that is, in these cases there is no probabilistic sample that would justify the use of inferential statistics procedures. However, OECD computes standard errors, replicate weights and plausible values, arguing that omissions in census administration can lead to the set of test-takers being treated as a probabilistic sample of the population (OECD, 2014, p. 140). This argument is untenable unless it is demonstrated that omission was a random event, which has not been done in this case.

Additional information

Notes on contributors

César Guadalupe

César Guadalupe is Senior Lecturer-researcher at the Department of Social and Political Sciences, Universidad del Pacífico (Lima, Peru). He holds bachelor’s degree and a Licenciatura in Sociology (Pontificia Universidad Católica del Perú), an MA in Social and Political Thought (University of Sussex) and a Doctorate of Education (University of Sussex). He is experienced in the field of education information systems, including both ‘traditional’ statistics and the measurement of learning outcomes. Prior to joining Universidad del Pacífico in 2012, he served at the UNESCO Institute for Statistics as head of the Learning Outcomes Section and, previously, as Regional Advisor for Latin America and the Caribbean. He was also a non-resident fellow of the Brookings Institution between 2013 and 2016. Since 2014, he has been a member of the Peruvian National Education Council, and was elected its Chair for the 2017–2020 period. A detailed profile and list of publications is available at: https://www.mendeley.com/profiles/cesar-guadalupe/