Cross-cultural comparability and validity of metacognitive knowledge in reading in PISA 2009: a comparison of two scoring methods

ABSTRACT

Accurate measurement of metacognitive knowledge in reading is important. Different instruments and scoring methods have been proposed but not systematically compared for their measurement comparability across cultures and validity. Given student data from 34 OECD countries in the Programme for International Student Assessment (PISA) in 2009, we compared two scoring methods for metacognitive knowledge in reading based on pair-wise comparisons of strategies and with conventional Likert-scale responses of selected items. Metacognitive knowledge scored with conventional Likert-scale responses demonstrated higher cross-cultural comparability than the pair-wise comparison method. Linked with reading competence, motivation and control strategy in reading, scores from the two scoring methods showed differential criterion validity, possibly related to the types of tasks (understanding and remembering versus summarising), item content (complexity and discrimination between preferred strategies in reading) and common method variance (e.g., individuals’ stable response style in rating scales). Theoretical and methodological implications are discussed..

KEYWORDS:

• Metacognitive knowledge
• reading
• cross-cultural comparability
• validity
• Programme for International Student Assessment (PISA)

Disclosure statement

No potential conflict of interest was reported by the authors.

Additional information

Notes on contributors

Ji Zhou

Ji Zhou has studied in Beijing Normal University and obtained her PhD in educational psychology in Ludwig-Maximilians-University München, Germany. Her research interest lies in cross-cultural assessment, learning motivation, metacognition, and the methods for large scale analysis.

Jia He

Jia He obtained her PhD in cross-cultural psychology in Tilburg University, the Netherlands. Her research includes cross-cultural assessment and advanced psychometric methods. She is interested in modern research methods such as item response theory-based scaling, structural equation modelling, multilevel analysis, and Bayesian statistics.

Dominique Lafontaine

Dominique Lafontaine is a Full Professor of Educational Science at the University of Liège and is director of the research centre ‘Analysis of systems and practices in education’ (aSPe). D. Lafontaine has a strong background in the field of reading literacy, comparative studies, teaching and learning processes, development of cognitive and non-cognitive instruments, effectiveness and equity of education systems. D. Lafontaine has been a member of the PISA-Reading expert group since 1998 and was a member of the Questionnaire expert group for PISA 2018.