Spoilt for choice? Issues around the use and comparability of optional exam questions

Abstract

For many years, question choice has been used in some UK public examinations, with students free to choose which questions they answer from a selection (within certain parameters). There has been little published research on choice of exam questions in recent years in the UK. In this article we distinguish different scenarios in which choice arises, explore the arguments for and against using optional questions, and exploit the item level data that has recently become available from on-screen marking of examinations to exemplify methods for investigating the (statistical) comparability of optional questions. We conclude that unless there is a very good reason for allowing question choice it should be avoided.

Keywords:

• Optional questions
• question choice
• comparability
• question difficulty

Notes

1. Daily Telegraph (7 December 2011) Exam boards: seminars costing £120 per head where teachers are told what will come up in exams. http://www.telegraph.co.uk/education/secondaryeducation/8940788/Exam-boards-seminars-costing-120-per-head-where-teachers-are-told-what-will-come-up-in-exams.html

2. In the US, the ‘rubric’ or ‘scoring rubric’ is what in England is called the mark scheme. What in England is called the rubric means the instructions to examinees about which questions to answer – e.g. ‘Answer five questions in total including at least one question from each of sections A, B and C’.

3. Though note that these studies used multiple choice and short answer questions, whereas optional questions tend to be higher mark questions.

4. In such mark schemes, often used at GCSE and A level for marking essays, the marks available for the question are split into levels or bands, with descriptors of performance within each band that can be applied to several or all questions on the paper. The marker decides first on which band the response lies in, and then on which mark point within the band.

5. OCR GCSE Religious Studies A, Paper B589/01.

6. OCR GCSE English Literature, Paper A664/02 Specimen.

7. General Certificate of Education Advanced levels (A levels) are subject-specific qualifications taken by large numbers of students in England, Wales and Northern Ireland, usually at 18 years old. In the 1990s AS stood for Advanced Supplementary and from 2000 it stood for Advanced Subsidiary. Both covered approximately half of the content of a full A level.

8. A grade boundary is the minimum mark required to achieve a particular grade. The grade boundaries for Exam 2 are shown by vertical dotted lines in Figure 2. Both Exam 1 and Exam 2 were components of assessments made up of several components. Any lack of comparability of optional questions would have less effect on the overall assessment grade than on the individual component grade.

9. In the terminology of Rubin (1976), this is the assumption that the missing data is ‘missing at random’ (MAR).