Assessing learning progress: validating a test score interpretation in the domain of sustainability management

ABSTRACT

This paper aims to provide evidence regarding the interpretation that test scores of a newly developed test instrument reflect learning progress in competence in sustainability management. As competence in sustainability management is conceptualised as mainly acquired through academic opportunities to learn (OTLs), students in courses with relevant academic OTLs (focus group) should display greater learning progress than those without (control group). Non-academic OTLs should not predict learning progress. 499 students were tested between winter term 2017/2018 and summer term 2018. We specify SEM with fixed effects for the courses and calculate linear contrasts between the focus and control group. In addition, we predict learning progress by self-rated academic and non-academic OTLs. Results show that for two of the tests, academic OTLs indeed predict learning progress on course level and non-academic OTLs do not. In sum, evidence suggests that scores for two of the tests reflect learning progress in competence in sustainability management.

KEYWORDS:

• Sustainability management
• higher education
• learning progress
• validity
• fixed effects model

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

1 The original German title of the project ‘Kompetenzmodellierung und -erfassung in Nachhaltigkeitsmanagement’ translates to ‘Modelling and Assessment of Sustainability Management Competence’.

2 The names of tests, questionnaires, scales, abbreviations thereof and item texts were all translated by the authors.

3 Performing the same model for KBA as for the sustainability-related tests (see method section) shows that that focus and control group do neither differ in pre-score (KBA.pre = −.036(.019), p = .062) nor in post-score (KBA.post = .052(.031), p = .09).

4 In the alternative models (no fixed effects but post-hoc correction of standard errors), self-reported academic OTLs predict learning progress for all tests. For all models, model fit is not acceptable (see ESM, Table A4).

5 In the alternative models (no fixed effects but post-hoc correction of standard errors), self-reported non-academic OTLs predict neither pre-knowledge nor learning progress in all of the tests. For all models, model fit is acceptable or better (see ESM, Table A5).

Additional information

Funding

This work was funded by the German Federal Ministry of Education and Research (BMBF) within the research program ‘Modeling and Measuring Competencies in Higher Education – Validation and Methodological Innovations’ under Grant 01PK15010.