![Sample our Education journals, sign in here to start your access, latest two full volumes FREE to you for 14 days]({"type":"image" , "src" : "/sda/5932/TOC-Subject-Sample-2021-Education.jpg"})
Abstract
The results of large-scale international assessments such as Programme for International Student Assessment (PISA) have attracted a considerable attention worldwide and are often used by policy-makers to support educational policies. To ensure that the published results represent the actual population, these surveys go through a thorough scrutiny to ensure their validity. One important issue that may affect the results is the occurrence of different participation rates across groups with heterogeneous average student scores. In this study, we illustrate how problems of representativeness of the samples may be corrected by the use of post-stratified weights. We use the case of Portugal, a country where we uncover a meaningful difference between the actual population and the PISA samples in terms of the distribution of students per grade and track of studies. We show that after post-stratification, the temporal evolution of the PISA scores is quite different from the one reported.
Acknowledgements
Support from Fundação para a Ciência e Tecnologia is gratefully acknowledged. We thank DGEEC – Portuguese Ministry of Education for the data provided. Suggestions from two anonymous referees are gratefully acknowledged.
Disclosure statement
No potential conflict of interest was reported by the authors.
Appendix
Since for PISA 2006, there are no observations relative to Lower Secondary Vocational courses and Upper Secondary Professional courses, it was necessary to define a method to assign a score to these students.
It was assumed that the proportional relationships between the scores of Upper Secondary Professional courses and the 9th grade and between the scores of Lower Secondary Vocational courses and 7th grade observed in public schools in 2006 were also verified in 2006.
Notes
1. Some policy-makers responses to the PISA results can be found at the OECD website: http://www.oecd.org/education/focus-world-reaction-to-pisa.htm.
3. Data already existed, but they were made available to the scientific community at this date.
4. See OECD, the PISA International Database at http://www.oecd.org/pisa/.
5. Actually, students’ ages fall between 15 years and 3 months old and 16 years and 3 months old.
6. If the rate of student participation falls short of 50% within a school, the whole school is withdrawn from the sample. For the validity of the whole survey, PISA sets a threshold of 85% for the school response and 80% for student response.
7. For further details on the design of PISA, see the PISA Technical Reports (OECD, Citation2012). A good description is provided in McGaw (Citation2008).
8. MISI is the acronym for ‘Information System of the Ministry of Education’.
9. Azores and Madeira account for 5.8% of the total 15-year-old Portuguese population in 2012. For 2009 and 2012 students from Azores and Madeira were excluded from PISA, too. For 2006 it was not possible to isolate this subsample. Excluding these students from the PISA sample never affects the final PISA scores by more than one point.
10. For 2000 and 2003, there are no statistical data with this level of disaggregation for 15-year olds.
11. The mean scores reported are always the means of the five plausible values calculated by PISA.
12. The same pattern is seen for students in the private system.
13. The effect of the response rates has been highlighted in the case of England by Micklewright, Schnepf, and Skinner (Citation2012).