A qualitative study of markers’ perceptions on onscreen marking in five subject areas

ABSTRACT

This paper reports on a qualitative study on markers’ perceptions of onscreen marking (OSM) in association with key influential factors of marking reliability. The study has made adaptations to an existing framework proposed by Black, Suto, and Bramley in 2011 for exploring issues related to influential factors of marking reliability in OSM contexts. Specifically, the study investigated those influential factors in the framework that can be manipulated and managed by the examination authority. The study involved 31 markers for a large-scale publication examination who were secondary school teachers from 5 subject areas. The major themes indicated that markers’ attitudes towards OSM might be potentially associated with the types of questions they marked. Further, the advantages and disadvantages of OSM as perceived by the markers were likely to be related to key influential factors of reliability pertaining to the quality control process and the usability of technology in the OSM system.

KEYWORDS:

• Onscreen marking (OSM)
• public examinations
• marking reliability
• quality control
• OSM technology

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes on contributors

Min Yang is an assistant professor in the Department of Curriculum and Instruction at The Education University of Hong Kong. She has researched into technology-enhanced feedback and assessment approaches to enhancing student learning.

Zi Yan is an assistant professor in the Department of Curriculum and Instruction at The Education University of Hong Kong. His main research interests are in educational and psychological assessment, especially the application of Rasch analysis.

David Coniam is a chair professor in the Department of Curriculum and Instruction at The Education University of Hong Kong, where he is a teacher educator, working with teachers in Hong Kong secondary schools. His main publication and research interests are in language assessment, language teaching methodology, and corpus linguistics.

ORCID

Zi Yan http://orcid.org/0000-0001-9305-884X

Notes

1 OSM quality control mechanisms used by the HKEAA for the HKDSE include: (a) A qualifying procedure, which uses standardisation scripts to determine a marker’s marking accuracy against prescribed answers and whether the marker may therefore qualify for marking. Qualifying involves: the training stage, where markers explore how the OSM system operates (e.g., viewing reference scripts that have been assigned “gold standard” grades); and the qualifying stage, where markers mark qualifying scripts with prescribed grades. (b) A double marking procedure, which is applied with subject panels where test papers include open-ended questions/items. Double marking involves each script being marked by two markers. If the discrepancy between the two markers’ marks exceeds a prescribed limit, a third (and sometimes a fourth) marker will be invoked.

2 The difference in marking reported by the geography teachers was because they were referring to OSM marking in different years. The method of marking in geography changed from marking by section to marking by question.

3 In their interviews, participants were asked to share their opinions about the logistical arrangements and the OSM environment. Nonetheless, participants did not report the two aspects as directly relevant to marking reliability. Therefore, this part of the data is not reported in the current paper.