The effects of rubrics on evaluative judgement: a randomised controlled experiment

Abstract

Rubrics have been suggested as a means to foster students’ evaluative judgement, the capacity to appraise their own work and that of others; however, empirical evidence of rubrics’ effectiveness is still emerging. This paper contributes findings from a randomised controlled experiment on the effect of rubrics on evaluative judgement. Participants were randomly assigned to one of two groups: a control group which evaluated peer-authored learning resources without the use of a rubric and an experiment group which carried out the evaluation using a three-item rubric based on (1) alignment with course content, (2) accuracy and (3) clarity. Both groups were asked to rate their confidence and provide comments to justify their scoring. The results showed a small effect size in increasing average agreement on the quality of learning resources in the experiment group. Analysis of comments reveals that criteria in and beyond the rubric guided participants’ ratings of quality. The study provides evidence of the impact of rubrics on students’ evaluative judgement and an example of how data-driven approaches and learning analytics can inform actionable design choices for embedding pedagogically supported strategies derived from the literature into actively operating educational technologies.

Keywords:

• Evaluative judgement
• rubrics
• peer moderation
• educational technologies
• higher education

Disclosure statement

No potential conflict of interest was reported by the authors.

Additional information

Notes on contributors

George Gyamfi

George Gyamfi is a PhD candidate in the School of Languages and Cultures at The University of Queensland. George aims to apply theories and concepts from the field of education in a novel and innovative way combined with insights from Information and Communication Technologies to make practical contributions to learning and research in higher education. His current research aims at investigating the effects of strategies for the development of students’ evaluative judgement using an adaptive online learning system.

Barbara E. Hanna

Dr Barbara E. Hanna is Senior Lecturer in French in the School of Languages and Cultures, The University of Queensland. She has worked on applications of Information and Communication Technologies to language and culture teaching; and on learner identity and agency.

Hassan Khosravi

Dr Hassan Khosravi is a Senior Lecturer In Data Science and Learning Analytics at The University of Queensland. In his research, he draws on theoretical insights driven from the learning sciences and human-computer interaction and exemplary techniques from the fields of machine learning and educational data mining to design, implement, validate and deliver socio-technical systems that improve student learning.