Treason and revenge: the emergence and continuation of ILSA contracting

ABSTRACT

OECD and IEA International Large-Scale Assessments (ILSAs) contractors have remained largely invisible despite playing a key role in the making of education assessment data. With the help of Bourdieu’s thinking, this paper analyses data generated through document analysis of ILSA Reports and in-depth interviews with staff working on ILSAs at the IEA, the OECD and ILSA contractors. The paper traces education assessment developments in the USA and at the IEA and the OECD since the 1980s, and points to the critical role of Education Testing Services (ETS). Although the selection of ILSA contractors ostensibly occurs through a globally competitive tendering process, this paper reveals the pressures and struggles that emerged when economic capital became available at the IEA and the OECD. The struggles reveal PISA was created as a competing project to TIMSS. The paper demonstrates how the social and cultural capitals that individuals acquired, and in particular trust, shaped who can and who cannot produce ILSA data today. Overall, the paper makes visible the ILSA data chefs and their recipes: ILSAs are not raw, they are cooked under pressure with personal and organizational tensions, struggles, conflicts, and emotional bonds.

KEYWORDS:

• ILSAs
• IEA
• OECD
• Bourdieu
• contracting
• network ethnography
• trust

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

1. This methodological section is written in the first person plural, as Addey carried out the data generating process with O’Connor (see our methodological paper, O’Connor and Addey, 2024).

2. For the OECD, this encompassed IALS in 1994 and 1998, ALL in ~ 2005, PIAAC between 2008-~2019, PISA 2000, 2003, 2006, 2009, 2012, 2015, 2018, 2021, and 2024, PISA-D in 2018, TALIS in 2008 and 2018, IELS in 2020*, and SSES in 2020. For IEA, this encompassed TIMSS 1995, 1999, 2003, 2007, 2011, 2015, and 2019, PIRLS 2001, 2006, 2011, and 2016, CIVED ~ 1998, ICCS 2009 and 2016, and ICILS 2013 and 2018.

3. There is a parallel between the ‘very scathing’ report that was used to discredit IEA’s management and the Puryear and Guthrie/Hansen 1995 reports that discredited UNESCO’s Institute for Statistics (UIS) (Addey, 2018 and Heyneman 1999). In both cases, U.S. government bodies steered reports that led to the discrediting organizations producing international education data, followed shortly after by substantial U.S. funding to develop new education data.

4. ETS and Westat already a had long-standing partnership, including their NAEP work.

5. Network A was the INES group that focused on Student Achievement Outcomes and was established in 1989. It was led by U.S. Department of Education/NCES. In 1995, the year Eugene Owen from NAEP took over the leadership of Network A, there was a pivotal Third General Assembly and the Network established that the OECD would develop a ‘new data strategy for collecting indicators on student outcomes’ (OECD/INES 1995: 129).

Additional information

Funding

The work was supported by the European Commission [Grant number Ref: GA-835479].

Notes on contributors

Camilla Addey

Camilla Addey is a Marie Curie Fellow at GEPS – the Globalisation, Education and Social Policies – research centre at the Department of Sociology of the Autonomous University of Barcelona, Spain. Formerly, Camilla was a Lecturer in Comparative and International Education at Teachers College, Columbia University (USA), and a researcher at Humboldt University in Berlin (Germany). Previously, Camilla worked in education at UNESCO headquarters. Her research interests are in international large-scale assessments in lower and middle income contexts, global educational policy, and education privatisations. Her current research project ILAINC, on the privatisation of international education assessment, can be followed on Facebook at ‘ILSA Inc. The ILSA industry’. She has published in Journal of Education Policy; Comparative Education; Globalization, Societies and Education; Compare: A Journal of Comparative and International Education; Critical Studies in Education; and Assessment in Education: Principles, Policy and Practice. Camilla is a director of the Laboratory of International Assessment Studies - http://international-assessments.org