Unattended consequences: how text responses alter alongside PISA’s mode change from 2012 to 2015

ABSTRACT

In 2015, the Programme for International Student Assessment (PISA) introduced multiple changes in its study design, the most extensive being the transition from paper- to computer-based assessment. We investigated the differences between German students’ text responses to eight reading items from the paper-based study in 2012 to text responses to the same items from the computer-based study in 2015. Two response features – information quantity and relevance proportion – were extracted by natural language processing techniques because they are crucial indicators for the response process. Showcasing potential differential relationships, we additionally examined gender differences. Modelling effects of the round of assessment, gender, and response correctness on the response features, we analysed responses from 15-year-olds and ninth-graders in Germany. Results revealed differences in the text responses between the rounds of assessment in that students included more information overall in 2015, and the proportions of relevance varied substantially across items. As the study investigated the mode change in PISA’s natural (not experimental) setting, the differences could mirror cohort trends or design changes. However, with the evidence reported, we conclude that the differences could indicate mode effects.

KEYWORDS:

• Computer-based assessment
• paper-based assessment
• open-ended text responses
• mode effect
• automatic processing

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes

1. Note that both sample sizes refer to the subsets of students for which text responses to the investigated items were available.

2. The conditional $R_c^2$, which includes both fixed and random effects, could not be computed for the non-Gaussian model. For making sure the model is worth investigating, we computed the Gaussian equivalent of the final model, which shows identical $R_m^2$ for all fixed effects and an $R_c^2$ of $.418$.

3. The brackets indicate 95% confidence intervals.

Additional information

Notes on contributors

Fabian Zehner

Fabian Zehner is a senior psychometrician at the German Institute for International Educational Research (DIPF), Centre for Technology Based Assessment (TBA). He received his PhD from the Technical University of Munich, Germany, for a dissertation centering around the automatic processing of open-ended text responses in educational large-scale assessments. His major research focuses include the application of natural language processing techniques and machine learning in assessments, assessment instruments employing innovative technologies, and the construction of psychological and educational assessments.

Frank Goldhammer

Frank Goldhammer is head of the Centre for Technology Based Assessment (TBA) at the German Institute for International Educational Research (DIPF) and professor for Educational and Psychological Assessment (Technology-Based Assessment and Instruction) at the Goethe University Frankfurt a. M. and the Centre for International Student Assessment (ZIB). His research interests include technology-based assessment (e.g., validation), the analysis of process data from cognitive assessments, modelling response times, as well as digital skills and motivational context variables.

Emily Lubaway

Emily Lubaway holds a Master of Education from the Technical University of Munich, Germany, in Research on Teaching and Learning. She is currently a Senior Research Assistant at Educational Testing Service (ETS) in the work-group that oversees the design and evaluation of international large-scale assessments, including the Programme for International Student Assessment (PISA) and the Programme for the International Assessment Adult Competencies (PIAAC).

Christine Sälzer

Christine Sälzer is a professor in education at the University of Stuttgart, Germany. Having been the national project manager for PISA in Germany over seven years, professor Sälzer focuses on making large scale student assessments accessible to future teachers. Her main research interests are large scale assessments, competence modelling, school absenteeism and students with special educational needs.