Assembled validity: rethinking Kane’s argument-based approach in the context of International Large-Scale Assessments (ILSAs)

ABSTRACT

Drawing on Kane’s argument-based approach to validity and Toulmin’s later work on cosmopolitanism and diversity, this paper asks whose validity arguments and evidence are being presented in International Large-Scale Assessments (ILSAs), where and when. With a case study of the OECD’s PISA for Development, we demonstrate that validity arguments are assembled, negotiated and transformed by the network of actors. We claim that the challenge of ILSAs is not to establish a single authoritative argument through the displacement of plural interpretations and uses. Instead, the tasks of an argument-based approach should be to create a democratic space in which legitimately diverse arguments and intentions can be recognized, considered, assembled and displayed.

KEYWORDS:

• Assembled validity
• validity
• validation
• International Large-Scale Assessments in Education (ILSAs)
• PISA for Development (PISA-D)
• OECD
• Kane

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes

1. Within the field of educational programme evaluation there were similar concerns at the time about the democratic representation of diverse stakeholders (e.g. Cronbach, 1988; E. House, 1980; E. R. House, 1990).

2. ANT has previously been applied to study the socio-material and technical aspects of assessment programmes (Addey, 2019b; Gorur, 2011; Gorur et al., 2019).

3. Toulmin’s (1958) work on argument is influential beyond assessment in studies of literature and rhetoric.

4. In a recent paper, Schaffner (2020) criticises Kane’s ‘rigid form of the Toulmin framework’ (p. 4). He considers the ‘quasidebate’ suggested by Kane’s application of Toulmin to be ‘at variance with the ways that scientific writers actually present arguments’ (p. 4).

Additional information

Notes on contributors

Camilla Addey

Camilla Addey is a Marie Curie Research Fellow at GEPS - the Globalisation, Education and Social Policies research centre - at the Department of Sociology at the Autonomous University of Barcelona, Spain. Her current research project looks into the privatisation of international learning assessment (Facebook page: ‘ILSA Inc. The ILSA industry’). Camilla is a co-Director of the Laboratory of International Assessment Studies. She is currently co-editing ‘Intimate Accounts of Education Policy Research: The Practice of Methods’ (Routledge 2021) with Nelli Piattoeva.

Bryan Maddox

Bryan Maddox is Associate Professor of Educational Assessment at the University of East Anglia, Professor II at the Centre for Educational Measurement at the University of Oslo, and Executive Director of Assessment Micro-Analytics Ltd. His research applies small-scale observational techniques to study assessment response processes and validity in large-scale educational assessments. His edited book on ‘International Large-Scale Assessments in Education: Insider Research Perspectives’ is published by Bloomsbury (2018).

Bruno D. Zumbo

Bruno D. Zumbo is Professor and Distinguished University Scholar, the Tier 1 Canada Research Chair in Psychometrics and Measurement, and the Paragon UBC Professor of Psychometrics & Measurement at the University of British Columbia. He leverages his mathematical sciences and psychometrics background and a long-standing interest in philosophical and conceptual matters in assessment and measurement to advance test validation.