Competitive comparison and PISA bragging rights: sub-national uses of the OECD's PISA in Canada and the USA

Abstract

Participation in the Organization for Economic Co-operation and Development's (OECD) Program for International Student Assessment (PISA) has continuously expanded: from 43 systems in 2000 to 65 systems in the 2012 cycle, with 71 signed up for PISA 2015. There also has been a growth in sub-national participation, expanding PISA's reach beyond the nation-state. This paper explores sub-national PISA participation in Canada and the USA, asking how PISA is being used within sub-national education policy spaces. We draw on analysis of documents and data from interviews with officials at sub-national, national, and international levels. Findings illustrate some of the diverse motivations and uses of PISA, providing insights into the effects of PISA at the sub-national scale. As such, we argue that competitive comparison in education has deepened through the enhanced granularity of international large-scale assessment data to new scales beyond the nation-state.

Keywords:

• OECD
• PISA
• sub-nation
• Canada
• USA
• policy
• assessment

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes

1. The OECD uses varying terms to describe PISA participants. On its website, the OECD generally refers to ‘economies’ and ‘benchmarking entities.’ In the PISA 2012 report, it refers to ‘countries,’ ‘economies,’ and ‘sub-national entities’ and also reports results for both ‘regional’ (sub-national) and ‘national’ levels. There is of course considerable ambiguity and complexity surrounding concepts of ‘country,’ ‘nation,’ ‘region,’ and ‘sub-nation,’ particularly for city-states and territories with disputed and/or changing political status. Often by avoiding ‘nation,’ the OECD is able to remain a neutral actor, circumventing political sensitivities and complexities surrounding these geo-political and historically-laden issues (Grek et al., 2009).

2. Analysis explored technical reports for 2000–2009 cycles. At the time of writing this article, the 2012 technical report has not yet been released.

3. Distinct from PISA's first four cycles, other international assessments, including TIMSS and PIRLS have included participating sub-nations in their international reports. For PISA, this change occurred in the 2012 cycle.

4. For all participants, there are three variables used to identify students and schools: the country identification variable (COUNTRY), a school ID (SCHOOLID), and student identification (STIDSTD). For adjudicated sub-nations, a fourth variable (SUBNATIO) is also used (OECD, 2009). For example, in PISA 2009, separate sub-national variables were included for Belgium (one value for the Flemish region, one for the French and German regions), all regions in Spain, and for Scotland in the UK (England, Northern Ireland, and Wales are assigned a single value; OECD, 2009).

5. The cost of PISA varies by country. All participating countries in PISA are required to pay the costs for implementing the program at the national level as well as contribute to the costs of international overhead. Sub-national costs vary depending on whether a sub-nation is adjudicated or nonadjudicated.

6. There are roughly 1000 schools that participated in PISA in Canada with a total of 21,000 15 year olds. In Ontario, 3699 students participated in PISA (EQAO, 2012).

7. The OECD representative reported conversations with ‘over a dozen states’ about PISA 2015 participation, and although participating states expressed an interest in continuing to participate in order to know whether progress was made or whether the Common Core State Standards were improving outcomes, at the time of the study, none of the states had formally committed to PISA 2015.

8. The PISA-Based Test for Schools was piloted in 2012 and first administered in 2013. It is too early to understand the impact this local assessment might have on sub-national participation. However, Canada and the USA participated in the pilot assessment, which may indicate some interest in using the test.