Insight from research across the world: examining student responses, assessment practices, and feedback engagement

In this first regular issue of 2024, the original articles span from international large-scale studies to students in the classroom.

The first article, by Steinmann et al. (2024), investigates patterns in student questionnaire responses in the TIMSS 2019 study. More specifically, the researchers examined which students are more likely to respond inconsistently to mixed-worded questionnaire scales, and which country samples have larger shares of inconsistent respondents. At the student level, Steinmann et al. (2024) investigated four predictor variables of inconsistent responding, separately and in a joint model: 1) mathematics achievement, 2) student age, 3) language at home, and 4) gender. Across the different models, countries, and grade levels, mathematics achievement stood out as the strongest and most consistent predictor of inconsistent responding. Interestingly, it was also found that girls in many countries responded more consistently than boys and that students speaking the test language at home are more likely to respond consistently. The authors discuss interpretations of the data noting that the design of the study does not allow for directly testing their interpretation that lack-of-skills seems to be the explanation and not students being careless when responding.

Such interpretations were also offered in similar studies investigating the validity of the PISA questionnaire data, where it was demonstrated that low-performing students on PISA science achievement did not report consistently their learning strategies measured in the PISA student questionnaire (Hopfenbeck & Maul, 2011). Further interviews confirmed some of the students did not even understand the items the way the test-developer intended. The study by Steinmann et al. (2024) raises the need for further studies investigating International Large Scale Assessment studies and inconsistency in student response patterns, with both quantitative and mixed methods, particularly considering the large-scale impact these studies and results have globally, both on assessment policy and practice.

In the second article, Chambers et al. (2024) outline how comparative judgement can be used as part of the moderation process when schools have non-exam assessments. Although most of qualifications completed as part of secondary school education (age 16–18) in England are assessed via exams marked by an awarding body, a few qualifications comprise non-exam assessments, internal assessments, such as portfolios, practical demonstrations of skills and performances marked by teachers. Teacher assessments are known for having challenges around reliabilities and standards globally. In England, a moderation process with expert judgers checking sample of teachers’ assessment is taking place to secure standards across schools (Gill, 2015). The research team examined whether Comparative Judgement (CJ) could be used as part of this moderation process, by running simulations on datasets. More specifically, they investigated the effects of CJ design parameters on the CJ estimates of script quality and how to assign moderator marks after the CJ procedure. The findings showed that certain design parameters had substantial effects on reliability and suggested minimum values for CJ protocols.

They conclude that the use of CJ for moderation has the potential through technological advances to shape a more efficient moderation process, although they acknowledge their current study did not include any humans and thus effects might differ in a real-life empirical study.

In the third article, Gamlem et al. (2024) investigates the perceptions and argumentations for grading among Norwegian teachers when they assess English as a Foreign Language. The research team examined possible gender differences with respect to how teachers graded their students and how teachers argued and reasoned about their decisions for the grade. Teachers from three schools graded a total of 182 assessments, more specifically essays in English as a Foreign Language, using a 26-point Grading Scale (26-PGS). It was found that female students generally received higher grades compared to male students. These results align with previous research indicating that girls tend to receive higher grades than boys when their performance is assessed by classroom teachers, as teachers often employ subjective criteria in grading, resulting in variability of grading practices both within individual schools and across different schools.

The final paper is authored by Nash and Thomas (2024) on the topic of feedback. The importance of feedback to students for their future learning is well known and has been extensively researched (Hattie & Timperley, 2007; Panadero, 2023; Winstone & Nash, 2023). Most studies have taken a quantitative approach looking at effect sizes or surveyed teachers’ and students’ perception and use of feedback. The current study by Nash and Thomas is the first to our knowledge applying linguistic text analysis to examine how students reflect on their feedback experiences. This novel approach and the study itself offer insight into how students act and engage with the written feedback they are provided by their teachers and are therefore an important contribution to our field. The researchers analysed records from 760 undergraduate single- or joint-honours psychology students at Aston University in the US, who commenced their first year of study between 2017 and 2019. As part of a compulsory introductory social psychology module, students were asked to complete coursework assignments on the topic of how to engage with feedback. A total of three reflective writing samples for each student were analysed by the researchers. More specifically, the research team examined whether certain linguistic features of students’ written reflections in the University’s Blackboard LMS, which students wrote over the course of 1 year, on their experience of receiving and using feedback, would be associated with their track record of accessing vs. ignoring their instructors’ feedback. Nash and Thomas (2024) did not find meaningful associations between feedback-accessing and students’ language use but instead demonstrated in their analysis that students with lower grades have a greater tendency to ignore feedback. Similarly, the analysis of the language revealed that students with lower grades tended to focus relatively more on the past or present in their written reflections than on the future. The main goal of the study was to investigate whether students’ written reflections on feedback experiences could reveal any linguistic cues which would be associated with those students’ tendencies to access their teachers’ feedback in the electronic LMS system. Although no evidence of such association was found in this study, the idea of using language analysis to further enhance our understanding of feedback literacy, use and application, should be encouraged. Moreover, as the authors rightly state:

[…]in order to establish valid and robust theoretical accounts of feedback literacy, it is important that nonsignificant empirical findings feature transparently in the literature alongside significant findings (see e.g. Patall, 2021).

With the ongoing replication crisis in psychology (Open Science Collaboration, 2015), we urge the assessment community to conduct more replication studies of past studies both in new contexts and with new measures and to report also non-significant findings. The latter is particularly important in an era where researchers and educators are pressured to focus upon ‘what works’ and trust in research is under constant threat by some political leaders globally. Only through high-quality research with openness and integrity can we continue to serve our global education community.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Notes on contributors

Therese Hopfenbeck

Dr. Therese Hopfenbeck is Professor of Educational Assessment, the Director of the Assessment and Evaluation Research Centre at the Faculty of Education, University of Melbourne, Visiting Fellow at Kellogg College, University of Oxford and Adjunct Professor of The Norwegian University of Science and Technology.