![Sample our Education journals, sign in here to start your access, latest two full volumes FREE to you for 14 days]({"type":"image" , "src" : "/sda/5932/TOC-Subject-Sample-2021-Education.jpg"})
Abstract
The Programme for International Student Assessment (PISA) is an important cross-national study of 15-year-olds’ academic knowledge and skills. Educationalists and public policymakers eagerly await the tri-annual results, with particular interest in whether their country has moved up or slid down the international rankings, as compared to earlier rounds. In 2015 a major change was implemented in PISA, with the introduction of computer-based assessment. This has the potential to reduce comparability of PISA test scores across countries and over time. We investigate this issue using PISA 2015 field trial data for three countries: Germany, Sweden, and Ireland. We show how, if left unaccounted for, the change to computer-based testing could limit the comparability of PISA test scores. We then describe the methodology the study organisers have used to account for such mode effects. Our key conclusion is that although the adjustment made is unlikely to overcome all the potential challenges of switching to computer-based tests, it represents an improvement over the alternative of making no adjustment at all.
Notes
1. Of the PISA 2015 countries, 15 completed the paper test and 58 the computer test. The paper-based countries were Albania, Algeria, Argentina, Georgia, Indonesia, Jordan, Kazakhstan, Kosovo, Lebanon, Macedonia, Malta, Moldova, Romania, Trinidad and Tobago, and Viet Nam.
2. PISA 2015 did not use a ‘computer-adaptive’ assessment design.
3. In PISA 2012, as the paper test was typically conducted in the morning and the computer test in the afternoon, Jerrim (Citation2016) could not rule out fatigue or test motivation as possible confounding explanations for the difference in paper and computer scores. Likewise, differences could have been due to the specific test questions posed, rather than being an effect of assessment mode per se.
4. By ‘independent’, we are referring to outside the international consortium responsible for delivering PISA 2015. The Educational Testing Service (ETS), as part of the international consortium, has also investigated the issue of mode effects (ETS, Citation2015).
5. While convenience samples were drawn, the field trial did attempt to mirror the population, through use of stratifying variables.
6. In contrast, pupils in Group 3 were assigned new science questions never administered as part of PISA before, and have therefore been excluded from our analysis.
7. We have excluded one German school from the analysis, where only computer-based testing was used in the field trial. As randomisation of pupils occurs within schools, this does not pose a threat to the internal validity of our results.
8. Further information is available upon request.
9. Pupils answered these questions after completing the test. Their responses could therefore be affected by assessment mode, e.g. pupils more likely to report internet access and computer use if they had just completed the CBA version of the test.
10. The few partial credit items were coded as zero for any incorrect or partially correct answer and one for fully correct responses. If a child did not reach or respond to an item, they were awarded a zero for the question.
11. The results based upon differences in average scores between groups are very similar to those presented, and are available upon request.
12. It has not been possible to investigate heterogeneity for other groups (e.g. by socio-economic status) as the field trial did not collect this information for both the PBA and CBA groups.
13. When estimating these quantile regression models, we also chose to exclude the control variables (the vectors D, B and μj in Equation [1]). Hence all estimates refer to unconditional differences at each achievement percentile.
14. Average PISA maths scores fell by eight points in Germany between 2012 and 2015 and three points in Ireland. In contrast, the average in Sweden rose by 16 points.
15. Average reading scores were broadly stable in Ireland and Germany between 2012 and 2015. In Sweden they increased by 17 points, taking the average back to the 2009 level. However, in the case of Ireland, overall performance in reading masks changes in the performance of boys and girls, which may be related to mode.
16. Comparing PISA 2015 to PISA 2012 results, average science scores fell in Ireland by 17 points and in Germany by 15 points. In contrast, they rose in Sweden by eight points.
17. In Ireland, comparing PISA 2015 to PISA 2012 results, a convergence in scores was observed for girls and boys, average reading scores for girls fell in Ireland by 11 points, and average reading scores for boys in Ireland increased by six points.
18. Test questions where δi ≠ 0 contribute to measurement precision, but do not provide information about the location of pupils on the PISA scale.
19. Indeed, even more fundamentally, one may question whether any educational (or psychological) assessment can be of the measurement kind (see e.g. Michell, Citation2008).
20. However, we are unable to comment upon any residual impact upon other key statistics, such as the distribution of performance (standard deviation, percentiles) or co-variation with demographic characteristics such as socio-economic status.