What's going on? Analysing visual data to understand context-based decision-making processes

Abstract

This brief research report gives an overview of a visually based methodology that has recently been used to shed light on situated decision-making in a high stakes educational assessment environment. Looking carefully at the feedback interactions taking place between senior and less senior examiners, the project aim was to gain insight into the nature of feedback given and the influences on the formation of such feedback. Given the high stakes’ consequences of the interactions taking place, technology was able to support the research process by allowing video data about examiners' practices to be gathered without intruding into the outcomes being observed. This report describes one approach that was used to analyse video data in order to address the initial research question. The video data allowed researchers to gain insights into how examiners monitor marking quality. These insights included both the frequency of monitoring behaviours and the processes underpinning them. Analyses showed a three-stage Evaluation–Diagnosis–Feedback process. Being able to isolate and reflect on individual behaviours allowed the actors to give insights into their actions, giving voice to the inherent complexities whilst not interfering with such processes.

Keywords:

• assessment
• examinations
• decision-making
• visual analysis
• feedback

Notes

‘Awarding bodies’ are organizations in the UK that are recognized by national regulators as being fit to award qualifications.

Advanced GCE is usually studied over a two-year period and is widely recognized in England, Wales and Northern Ireland as being the standard entry qualification for assessing the suitability of applicants for academic courses in UK universities.