Developing evaluative judgement for a time of generative artificial intelligence

Figures & data

Figure 1. Three assessment strategies for developing evaluative judgement.

Table 1. Assessment strategies that develop evaluative judgements in a time of generative AI (categories replicated from Tai et al. (2018)).

Strategy	Current means to develop evaluative judgement (and illustrative reference)	Developing evaluative judgement with respect to:
Self-assessment	Self-assessment allows the opportunity for students to examine their own work against external criteria, which is a useful means to develop evaluative judgement. Ideally, these same criteria, whether in rubrics or other form, will span units, allowing the students to trace their developing understanding of how to identify quality in their own work (e.g. McIver and Murphy 2023).	Assessing generative AI outputs Self-assessment practices can be explicitly expanded to consider how to identify quality when working with generative AI. This means developing: a) evaluative judgements about the quality of the outputs; and b) evaluative judgements about how well these outputs have been integrated into the task, possibly employing external criteria such as rubrics. Assessing generative AI processes Self-assessment tasks can also include students’ evaluative judgements about their processes pertaining to generative AI – e.g. how iterative prompts were utilised and the quality of these strategies. generative AI assessment of student evaluative judgement The generative AI can directly assess the student’s work and then the student can compare their own judgement with the generative AI’s.
Peer assessment or peer review	Peer assessment allows students to review others’ work or performance, articulating evaluative judgements on how they address the relevant criteria or standards and how the work could be improved. This may be a reciprocal process where students also receive comments back on their own work, or may be engaged in purely for the practise of making and articulating judgements (e.g. Chen et al. 2022).	Assessing generative AI outputs Peers can jointly review each other’s evaluative judgements of generative AI outputs. Again, this is both: a) appraisal of the outputs; and b) appraisal of the student’s ability to integrate the AI into the task vis-à-vis external criteria. Assessing generative AI processes Peers can jointly review each other’s generative AI processes that underpin an assessment task. Peers can work together to iteratively develop prompts and judge their quality in relation to the outputs. generative AI assessment of student evaluative judgement A peer review role could develop into being a ‘human in the loop’ who either approves or corrects a range of generative AI judgements on others’ work. It still requires students to interact with work and make judgements independently of the generative AI tool, as well as with it.
Feedback	Students solicit and receive feedback inputs from others that focus on the evaluative judgements the learner makes about their own work, rather than the substantive features of that work. They compare their own judgements with those of others (e.g. De Mello Heredia, Henderson, and Phillips 2023).	Assessing generative AI outputs Students can iteratively request feedback outputs from generative AI about their work and compare these texts with others’ judgements, including their own. Thus, the student’s evaluative judgement is built around generative AI’s capability to provide useful information about own work, about the student’s own feedback processes, and about the quality of the work itself. Assessing generative AI processes Students can reflect on the variety of feedback inputs and outputs they have garnered from generative AI and start to form explicit judgements about their own processes in working with generative AI. In addition, students can solicit feedback outputs from generative AI about how they have worked with generative AI and compare these outputs with others’ judgements, including their own. generative AI assessment of student evaluative judgement With generative AI, feedback and self-assessment strategies become linked: the generative AI can directly assess the student’s work, and the student compares their own judgement with the generative AI’s.
Rubrics	Students and/or teachers construct rubrics as a way to articulate and explore relevant quality standards. Students use rubrics as a scaffold to support their evaluative judgement when undertaking self- and peer-assessment (e.g. Gyamfi, Hanna, and Khosravi 2022).	Assessing generative AI outputs When using outputs from generative AI as part of their assessed work, students can use rubrics to evaluate the quality of this supplemented work and identify aspects of the work that require human intervention to improve. Thus, evaluative judgement is built about the generative AI outputs and the work itself. A caution is that analytical rubrics may be limited when it comes to holistic judgements about quality (Sadler 1989). Assessing generative AI processes Peers can employ a rubric to jointly review each other’s generative AI processes that underpin an assessment task. Where there is a difference in judgement, they engage in a dialogue with generative AI and/or others to decide which information to trust. The rubric itself may focus on assessing generative AI processes, both skills and ethical judgements. generative AI assessment of student evaluative judgement The student can ask the generative AI to assess how well the student has used the rubric; the student and the generative AI may also assess the rubric itself and the student can compare comments about its qualities. Students can construct rubrics with generative AI as a means of exploring relevant quality standards.
Exemplars	Students use exemplars from a variety of sources to enable them to identify features of good work of the type they are seeking to produce. They compare their own subsequent work with the exemplars (e.g. Chong 2021).	Assessing generative AI outputs Students can use exemplars generated from a variety of generative AI outputs to enable them to identify features of good work of the type they are seeking to produce. They compare their own subsequent work with these exemplars and then seek further exemplars from the AI source. This iteratively develops evaluative judgement through these comparisons. They may need to draw on other exemplars from credible sources for calibration. Assessing generative AI processes In the above exemplar process, students can review how well their processes have worked in generating exemplars. Students could also be provided with exemplars of how to work with generative AI within tasks. generative AI assessment of student evaluative judgement Students prompt the generative AI to assess their judgements about exemplars and these can be compared to peers, teachers and own assessments.

Tai, J., R. Ajjawi, D. Boud, P. Dawson, and E. Panadero. 2018. “Developing Evaluative Judgement: Enabling Students to Make Decisions about the Quality of Work.” Higher Education 76 (3): 467–481. doi:10.1007/s10734-017-0220-3.

 Web of Science ®Google Scholar

McIver, S., and B. Murphy. 2023. “Self-Assessment and What Happens over Time: Student and Staff Perspectives, Expectations and Outcomes.” Active Learning in Higher Education 24 (2): 207–219. doi:10.1177/14697874211054755.

 Web of Science ®Google Scholar

Chen, L., S. Howitt, D. Higgins, and S. Murray. 2022. “Students’ Use of Evaluative Judgement in an Online Peer Learning Community.” Assessment & Evaluation in Higher Education 47 (4): 493–506. doi:10.1080/02602938.2021.1933378.

 Web of Science ®Google Scholar

De Mello Heredia, J., M. Henderson, and M. Phillips. 2023. “Using Video Feedback to Support Students.” Evaluative Judgement Society for Information Technology & Teacher Education International Conference 2023, New Orleans, LA. https://www.learntechlib.org/p/222090

 Google Scholar

Gyamfi, G., B. E. Hanna, and H. Khosravi. 2022. “The Effects of Rubrics on Evaluative Judgement: A Randomised Controlled Experiment.” Assessment & Evaluation in Higher Education 47 (1): 126–143. doi:10.1080/02602938.2021.1887081.

 Web of Science ®Google Scholar

Sadler, D. R. 1989. “Formative Assessment and the Design of Instructional Systems.” Instructional Science 18 (2): 119–144. doi:10.1007/BF00117714.

 Web of Science ®Google Scholar

Chong, S. W. 2021. “University Students’ Perceptions towards Using Exemplars Dialogically to Develop Evaluative Judgement: The Case of a High-Stakes Language Test.” Asian-Pacific Journal of Second and Foreign Language Education 6 (1): 12. doi:10.1186/s40862-021-00115-4.

 Web of Science ®Google Scholar