What do GCSE examiners think of ‘thinking aloud’? Findings from an exploratory study

Abstract

Background: ‘Thinking aloud’ is a well-established method of data collection in education, assessment, and other fields of research. However, while many researchers have reported their views on its usage, the first-hand experiences of research participants have received less attention.

Purpose: The aim of this exploratory study was to obtain the perspectives of GCSE examiners on thinking aloud as a means of investigating the cognitive strategies they use to mark examinations.

Sample: The study focused on the experimental marking of two contrasting GCSE examination papers from 2004: mathematics and business studies. Twelve experienced examiners (six for each subject) participated in the study as paid volunteers.

Design and methods: All examiners were individually asked to think aloud while concurrently marking up to five examination scripts each. Afterwards, they participated in one-to-one, semi-structured interviews about their experiences of the method. These interviews were recorded, transcribed and analysed qualitatively.

Results: There were no clear differences between the mathematics and business studies examiners. However, views on the validity of thinking aloud, and advantages and disadvantages, varied considerably among individuals, possibly reflecting diverse personal marking techniques within each subject. Examiners varied in the kinds of questions they found it hardest to think aloud about.

Conclusions: The findings have implications for further research and marking practice. They challenge two contrasting views reported in the literature: (1) that the simplest judgements are always hardest to verbalise; and (2) that it is always hardest to think aloud when engaging in complex thought processing. This indicates apparent variation in validity of the think aloud method among individuals. However, thinking aloud may also be useful in the training of new examiners.

Keywords:

• thinking aloud
• assessment
• examiners
• verbal protocols
• GCSE

Acknowledgements

This research is based on examinations administered by Oxford, Cambridge and RSA examinations (OCR) and was funded by Cambridge Assessment (the brand-name of the University of Cambridge Local Examinations Syndicate (UCLES)). The opinions expressed in this paper are those of the authors and are not to be taken as the opinions of Cambridge Assessment or OCR.

Notes

1.Candidates can study for GCSEs in a wide variety of subjects, which are administered by independent Awarding Bodies. GCSEs are assessed predominantly through examinations marked by professional examiners who are generally experienced school teachers. However, teachers are not allowed to mark the examination scripts of their own students.

2.Some GCSE examinations are divided into two or more tiers and candidates are assigned to a tier according to their ability in the subject. For example, the mathematics question paper was from an assessment for which candidates were divided into three tiers. The foundation tier assessed grades G to D, the intermediate tier assessed grades E to B and the higher tier assessed grades C to A∗. Candidates could only be assessed in one tier. Generally speaking, there are two styles of mark schemes for GCSE examination questions: points-based mark schemes and levels-based mark schemes. Points-based mark schemes provide guidelines to examiners regarding how many marks to credit candidates for the tasks that they accomplish correctly. Level-based marks schemes divide the marks available into bands, for example, 1–3 marks, 4–6 marks and 7–9 marks. Each band of marks is accompanied by a description of the types of responses that should be credited marks within the given band. The examiners use a principle of ‘best fit’ when they are applying the mark scheme. Initially, the examiners identify a description that best fits a candidate's answer, and secondly, they select the appropriate mark from within the band depending on the merit of the candidate's response. This two-stage process is repeated for each response.