Are questionnaire scales which measure non-cognitive constructs suitable as school effectiveness criteria? A measurement invariance analysis

ABSTRACT

This study aimed at examining the suitability of questionnaire instruments commonly used in large-scale assessments for measuring non-cognitive school effectiveness criteria. It focused on questions of reliability and validity for capturing changes in students within schools across time and the instruments’ sensitivity to school effects. The aim was also to propose an approach for analyzing measurement invariance across levels and time simultaneously. The study used longitudinal data from the KESS (Kompetenzen und Einstellungen von Schülerinnen und Schülern [competencies and attitudes of students]) study, conducted in Hamburg, Germany. The sample is comprised of 17,926 students in 189 secondary schools. Data were collected three or four times. The results of the analyses show that all investigated scales are suitable in terms of reliability and validity. However, only the scale interest in mathematics is sensitive to school effects and therefore suitable as a school effectiveness criterion.

KEYWORDS:

• School effectiveness criteria
• multilevel measurement invariance
• longitudinal measurement invariance
• non-cognitive constructs

Acknowledgements

This paper uses data from the longitudinal studies LAU and/or KESS. Both data sets were generated by the Free and Hanseatic City of Hamburg through the Ministry of Schools and Vocational Training between 1995 and 2012 and have been provided to the MILES scientific consortium (Methodological Issues in Longitudinal Educational Studies) for a limited period with the aim of conducting in-depth examinations of scientific questions. MILES is coordinated by the Leibniz Institute for Science and Mathematics Education (IPN).

Disclosure statement

No potential conflict of interest was reported by the authors.

Correction Statement

This article has been republished with minor changes. These changes do not impact the academic content of the article.

Notes

1 For a more detailed description of the study, see Bos et al. (2009), Bos and Gröhlich (2010), Bos and Pietsch (2006), and Vieluf et al. (2011, 2014).

2 While the academic track is intended to prepare students for university, the vocational track prepares students for vocational training. Comprehensive schools are a joint form of schools in which both tracks are offered.

3 Migration background is defined as having at least one parent born abroad.

4 One item was excluded from the analyses because an exploratory factor analysis showed that it loaded on both subscales.

5 Student level: gender, migration background, and HISEI; class/school level: the proportion of girls in class/school, the proportion of students with a migration background in class/school, and the average HISEI in class/school).

6 Thresholds only exist at the school level, so that scalar invariance over time can only be examined for the school level in a two-level model.

Additional information

Funding

This study was supported by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation), as part of the project “Modeling Educational Effectiveness and Equity at the School Level”, Project HA 5050/7-1.

Notes on contributors

Luisa Grützmacher

Luisa Grützmacher is a doctoral candidate within the Educational Quality and Evaluation Department at the DIPF | Leibniz Institute for Research and Information in Education. Her research interests include educational effectiveness research and in particular the suitability of school effectiveness criteria and the consistency of school effects.

Svenja Vieluf

Svenja Vieluf holds a PhD in Psychology and is currently a post-doc researcher within the Educational Quality and Evaluation Department at the DIPF | Leibniz Institute for Research and Information in Education. She has been involved in instrument development and data analysis for several national and international large-scale surveys (i.e., OECD studies PISA and TALIS). Currently, her research is focused on teaching and learning in heterogeneous classrooms, transcultural learning, and educational justice.

Johannes Hartig

Johannes Hartig is professor of educational measurement and head of the unit Educational Measurement in the Department of Educational Quality and Evaluation at the DIPF | Leibniz Institute for Research and Information in Education in Frankfurt, Germany. His research interests focus on psychometric models in competence assessments, including models for measuring effects of learning opportunities.