Who makes it into PISA? Understanding the impact of PISA sample eligibility using Turkey as a case study (PISA 2003–PISA 2012)

ABSTRACT

New evidence presented in this paper shows that existing accounts of Programme for International Student Assessment (PISA) in developing countries underestimate both progress and inequality because they do not take into account the large proportion of 15–16-year olds who are ineligible for the PISA sampling frame, primarily due to dropout or delay. In such countries, who makes it into PISA (sample eligibility) heavily influences representivity and therefore the interpretation of PISA results. We use PISA in Turkey 2003–2012 as a case study to show how combining measures of access (from the Turkish Demographic and Health Surveys) and learning outcomes (from PISA) can account for changes in the underlying population. We find the percentage of 15–16-year olds reaching Level 2 in PISA is up to twice as large as that reflected in official PISA reports. These findings have strong implications for inter-country and inter-temporal comparisons using PISA, particularly for developing countries.

KEYWORDS:

• PISA
• sample eligibility
• access
• Turkey
• representivity

Disclosure statement

No potential conflict of interest was reported by the author.

Notes

1. The sampling frame for the PISA assessment covers only those students that are aged 15 years and 3 (completed) months to 16 years and 2 (completed) months who are currently enrolled in Grade 7 or higher (OECD, 2005, p. 46). As is discussed later in the paper, using both 15- and 16-year olds increases the sample size (and therefore decreases the standard errors) when using the household survey data.

2. The lower estimate reflects the OECD’s own ‘Coverage Index 3’ for Turkey 2003 (OECD, 2004, p. 321), while the upper estimate is taken from the Turkish Demographic and Health Survey (DHS) data for 2003. Both data sources are discussed further in the paper.

3. For ease of reading, throughout this paper I use ‘sample eligibility’ and ‘sample coverage’ interchangeably. In the technical terms used by the OECD, these are marginally different because sample coverage is sample eligibility minus legitimate exclusions. These exclusions might be school-level exclusions (for inaccessibly remote schools), or for students that are intellectually disabled, for example. These exclusions are almost always less than 5% in developing countries (OECD, 2012b: 149). Thus, wherever the paper refers to coverage, it should technically be read as ‘coverage after exclusions among those eligible’.

4. I have confirmed this with OECD officials familiar with PISA sampling and the reports in question (Miyak oIkeda, personal communicati on, 22 June 2016).

5. The exact number of pages that the word ‘coverage’ can be found relative to the total number of pages in the report for each year are as follows: 2003 (8/471), 2006 (8/390), 2009 (10/276), 2012 (11/564) and 2015 (160/494).

6. Access was formally requested and granted under Project Number: 906,922.

7. Formal access was granted to both data sets on the 17 May 2016.

Additional information

Funding

This work was supported by the Thomas J Alexander Fellowship funded by the Open Society Foundation (2016).

Notes on contributors

Nic Spaull

Nic Spaull is currently a Senior Research Fellow at the Research on Socioeconomic Policy (RESEP) group at Stellenbosch University in South Africa. He has spent time as a research fellow at both the OECD in Paris and Stanford University in the United States. Nic has a PhD in economics and has published numerous journal articles on assessment, accountability, literacy and education policy in South Africa. Nic is currently working on the Funda Wande project which is developing a certificate to teach Foundation Phase teachers how to teach early grade reading in African languages.