School self-evaluation instruments and cognitive validity. Do items capture what they intend to?

ABSTRACT

School self-evaluation (SSE) often makes use of questionnaires in order to sketch a picture of the school. How respondents cognitively process questionnaire items determines the validity of SSE results. Still, one readily assumes that respondents interpret and answer items as intended by the instrument developer (referred to as cognitive validity), but it remains unclear whether they do. This study tested an exemplary SSE instrument by focusing on the extent to which SSE results are cognitively valid, and on the extent to which differences in cognitive validity can be attributed to respondents and/or items. Cognitive interviews with 20 participants made respondents’ answering processes manifest. Results show that, overall, fewer than 50% of respondents’ processes of interpreting and elaborating on items are cognitively valid. Cross-classified multilevel analyses indicate that various hierarchical levels, respondents and items, are significant in explaining differences in cognitive validity, but not for all stages of the answering process.

KEYWORDS:

• School self-evaluation
• cognitive validity
• questionnaires
• item design

Disclosure statement

No potential conflict of interest was reported by the authors.

Additional information

Notes on contributors

Jerich Faddar

Jerich Faddar is a PhD candidate at the Department of Training and Education Sciences (Faculty of Social Sciences) of the University of Antwerp and a member of the Edubron research unit. His current work focuses on methodological issues related to school self-evaluations.

Jan Vanhoof

Jan Vanhoof is Associate Professor on the staff of the Department of Training and Education Sciences of the University of Antwerp. He is a member of the Edubron research unit. His current research activities focus on school policy and quality assurance in general and on school self-evaluation and the use of data in education in particular.

Sven De Maeyer

Sven De Maeyer is Professor at the Department of Training and Education Sciences of the University of Antwerp. He is a member of the Edubron research unit. His research focuses on educational measurement and methodological issues in educational sciences.