Lost in translation: PISA experts, brokers, and marionettes

ABSTRACT

This paper explores how the OECD acts a broker of knowledge-making in the development of PISA, the most widely known International Large-Scale Assessment. Drawing on the work of Bandola-Gill, Grek, and Tichenor (2022) and analysing empirical data gathered through interviews with OECD staff and PISA contractors and experts, the paper analyses how the OECD brokers the making of comparative knowledge on learning outcomes. The paper does this by unpacking what counts as expertise in the making of PISA, and how the OECD manages and uses this expertise. Exploring the ways that international organizations broker experts and utilize expertise reveals how IOs first select experts and then delegate their expert-brokering role; they then seek to share the delegated process; and finally, take back expert-brokering and knowledge-production by creating, or decreeing, consensus. The paper also shows how expertise – beyond informing on a technical level – is used to shift scientific responsibilities and build global consensus.

KEYWORDS:

• International Organizations
• experts
• PISA
• knowledge production
• OECD

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

1. A study of power struggles in the PGB is well overdue, especially with the growth of PISA participation and the lack of lower and middle income countries with voting rights in the PGB.

1. One interviewee describes the different expertise that contractors provide: ‘When you put together an assessment, you have to have expertise in sampling. You have to have expertise in survey operations. You have to have expertise in knowing how to build frameworks and operationalize them so you get good solid measurement. You need to know how to translate and adapt these measures to work in comparable ways across languages and cultures. You also need to have methodological knowledge and experience in developing assessment designs and applying the various models. You also need a team with experience in developing the delivery platform, along with teams who have experience in processing and analyzing the data. So it’s an integration of all of these areas of expertise that are required to design, develop and conduct an assessment – and then you have to be able to manage it all and work together with countries and organizations like the OECD or IEA.’ Interviewee R, 2020

2. Director of the OECD Education and Skills Directorate, better known as the father of PISA.

3. This concept differs to Lubienski’s (2018) brokers or intermediators understood as intermediary organizations which seek to ‘connect research producers with users’ (2018: 160).

4. The TAG’s main focus is psychometrics.

5. The actors that the OECD brought together under the INES project to establish learning outcomes indicators, and which wrote data strategy that was used to develop PISA (see Morgan 2007).

6. INES, described by Grek and Ydesen (2021) as the precursor of PISA, was created in 1988 to gather, process and report international statistics and indicators on education (see Bottani 1996 for a detailed analysis of how the INES indicators were developed).

7. The Global Partnership of Education (GPE) emerged from the World Bank’s Fast Track Initiative (FTI). The GPE was established in response to growing criticism of FTI’s governance mechanisms – in particular, the dominance of the World Bank and donors in making decisions (Menashy 2019). The GPE is the largest multi-stakeholder educational partnership with members from the Global South and Global North. It was restructured as an equal, more just funding mechanism in line with the Paris Declaration (2005), the Accra Agenda for Education (2008) and the Busan Partnership for Effective Development Co-Operation (2011) for international aid to education.

8. The PGB includes 38 representatives from the participating governments (OECD Member States and Brazil) who have the right to vote. It also includes all other participating governments; these latter are observers and do not have voting rights.

9. This methodological section is written in the first person plural, we, as Camilla Addey carried out the data gathering process with Chloe O’Connor (see our methodological paper, O’Connor and Addey 2024).

10. For the OECD, this encompassed IALS in 1994 and 1998, ALL in ~ 2005, PIAAC between 2008-~2019, PISA 2000, 2003, 2006, 2009, 2012, 2015, 2018, 2021, and 2024, PISA-D in 2018, TALIS in 2008 and 2018, IELS in 2020*, and SSES in 2020. For IEA, this encompassed TIMSS 1995, 1999, 2003, 2007, 2011, 2015, and 2019, PIRLS 2001, 2006, 2011, and 2016, CIVED ~ 1998, ICCS 2009 and 2016, and ICILS 2013 and 2018.

11. These are IALS in 1994 and 1998, ALL in ~ 2005, PIAAC between 2008-~2019, PISA 2000, 2003, 2006, 2009, 2012, 2015, 2018, 2021, and 2024, PISA-D in 2018, TALIS in 2008 and 2018, IELS in 2020*, and SSES in 2020.

12. This is a private company contracted by the OECD to re-develop the background questionnaires of PISA-D.

13. These included, among others: actor characteristics (including expert selection); rationales for involvement; actor entries, moves and exits; constraints; market dynamics; regulative structures; cognitive frames; relationships; struggles and tensions (including management of expertise); and stories.

14. Educational Testing Services but known widely as ETS.

15. Australian Council for Educational Research but known widely as ACER.

16. In Addey (forthcoming, I question whether this autonomy was intended to take away decisional power from experts and national representatives (simultaneously involved at the IEA) and hand it over to a contractor whom the OECD could steer in more autonomy.

17. The PISA 2015 Report states ‘PISA is implemented within a framework established by the PISA Governing Board (PGB) which includes representation from all participating countries and economies at senior policy levels. The PGB establishes policy priorities and standards for developing indicators, for establishing assessment instruments, and for reporting results.

18. In other words, expertise vanishes as boundary practices appear and decisions are decreed.

19. This is a reference to George Orwell’s Animal Farm. What this sentence means is that although there are efforts to make all voices participate equally, some voices count more than others.

Additional information

Funding

This work was supported by the European Commission [GA-835479].

Notes on contributors

Camilla Addey

Camilla Addey is a Marie Curie Fellow at the Department of Sociology of the Autonomous University of Barcelona, Spain. Formerly, Camilla was a Lecturer in Comparative and International Education at Teachers College - Columbia University (USA), and a Research Fellow at Humboldt University in Berlin (Germany). Through the study of International Large-Scale Assessments, Camilla contributes to scholarship on the politics of education data, global education governance, global education policy, and education privatisations. She is co-founder and co-director of the Laboratory of International Assessment Studies.