The public understanding of error in educational assessment

Abstract

Evidence from recent research suggests that in the UK the public perception of errors in national examinations is that they are simply mistakes; events that are preventable. This perception predominates over the more sophisticated technical view that errors arise from many sources and create an inevitable variability in assessment outcomes. The public perception also seems to invest assessment grades and marks with the precision and accuracy of scientific measurements; a perception that does not sit easily with the academic and professional understanding that grades and marks are assigned rather than measured. However, growing numbers of successful challenges by students to their examination results present an interesting challenge for examination bodies. Such evidence could point to a number of possible causes. For example, there might be an increasing awareness among students of the uncertainty surrounding the grades assigned to their work; or their confidence in the capacity of such assessments to reflect the true quality of their performance may be decreasing. As the numbers of challenges increase year on year, there is growing consensus among assessment experts that public confidence needs to be strengthened. Clearly this may be achieved in a number of ways, e.g. by reducing the incidence of human errors and system breakdowns or by improving the public understanding of the assessment process. The latter tactic prompts calls for greater openness and transparency in all aspects of assessment design and process. However, such calls rarely touch on one of the most enduring dimensions of the problem: the public’s perception of precision and accuracy in educational assessments. This paper argues that this problem partly arises from a misuse of the term ‘measurement’ in educational assessment and that in addition to openness and transparency, any public understanding strategy should seek to reduce the misconceptions it causes.

Keywords:

• error
• educational assessment
• public understanding

Acknowledgement

I very much appreciated the comments of Hugh Morrison, Paul Black and Gordon Stanley on earlier drafts of this paper. However, responsibility for the positions I have taken rests entirely with me.

Notes

1. The research involved two sets of focus group interviews and the other 36 participants comprised teachers, students and examiners.

2. Ernst Gombrich later developed (1979) his idols vs ideals argument in a series of essays on values in the humanities, in which he criticised the seeming imperative to collect and analyse numerical data for every scholarly pursuit.

3. Civil Aviation Authority’s pass-mark for the Joint Aviation Regulations-Flight Crew Licensing (JAR-FCL) Theoretical Knowledge Examinations, see http://www.caa.co.uk/docs/175/srg_lts_LASORS2010_Section%20J.pdf