What makes AS marking reliable? An experiment with some stages from the standardisation process

Abstract

It is particularly important that GCSE and A‐level marking is valid and reliable as it affects the life chances of many young people in England. Current developments in marking technology are coinciding with potential changes in procedures to ensure valid and reliable marking. In this research the effectiveness of procedures to facilitate the reliability of marking were evaluated using a design with three interventions. Two of the interventions, the standardisation meeting and personal feedback, are similar to the conventional approach in GCSE and A‐level practice but within the constraints of a research study. The other intervention, pre‐written feedback, is how examples of marked scripts might be presented for part of examiner standardisation or training in other situations; it is also similar to the approach to feedback used in earlier research by Shaw and Baird et al. Each participating examiner experienced two of the three types of intervention. Marking data were gathered before and after the first intervention as well as after the second intervention. Inter‐examiner reliability was analysed using ANOVA models. Due to the different levels of inter‐examiner reliability at the beginning of the experiment it was hard to compare the effectiveness of the different interventions. Nevertheless, this research highlights that: (1) in line with previous research, standardisation meetings on their own are not particularly effective for experienced examiners; (2) as has been found in previous research, more experienced examiners were more lenient; (3) most combinations of interventions (combinations of standardisation processes – including the conventional approach) led to a decline in the size of the marking error; and (4) sometimes feedback (personal or pre‐written) was effective and sometimes it was not.

Keywords:

• inter‐examiner reliability
• GCE A‐levels
• examinations
• marking
• standardisation of marking

Acknowledgements

Thanks to members of the Research Division at Cambridge Assessment for their advice and support.

Notes

1. GCSE assessments are taken by nearly all 16‐year‐olds at the end of their compulsory schooling in England. The GCSE assessments are mostly examinations taken at the end of two years of study. Normally GCE A‐levels are taken by 18‐year‐olds. AS qualifications are intermediate qualifications between GCSEs and A‐levels. In addition to AS study and assessments, A2 study and assessments can be taken by students. The results from the AS and the A2 assessments are combined to give an A‐level result. The results of these assessments affect their life chances, including continuing with further study and obtaining a job. Amongst other qualifications, Oxford, Cambridge and RSA Examinations (OCR) provides A‐level, AS and GCSE qualifications in England. Cambridge Assessment is the parent group of OCR. Cambridge Assessment is a department of the University of Cambridge.

2. Sometimes these meetings and the associated process are called ‘co‐ordination’ meetings and processes but in the OCR ‘Instructions for Examiners June 2006’ they are called ‘standardisation’ meetings and processes; that terminology is used in this article.

3. The following information has not been included in the body of the report as it makes the text more difficult and complicated to read. The research was reported as one experiment, but it was also two smaller studies combined into one large experiment. The first of the two smaller studies was experimental groups 1 and 2. The second study was experimental groups 3 and 4. It was possible to combine the two studies into one experiment by using the same scripts, PE, PE∗ and time frame. It is important to note that the experiment constituted two small studies, as this was how the organisation of the research was explained to the examiners.