Threats to the professional understanding of assessment error

Abstract

A case study investigation was undertaken to identify threats to the professional understanding of assessment error which arise from accounts presented within the education press. Through a predominantly qualitative analysis of articles published in a leading education newspaper, during 2002 and 2003, it explored how assessment agencies in England were represented as responding to allegations of error. A number of threats to professional understanding were identified; in particular, the overarching threat that media reports may help to construct, and to maintain, a mythical image of assessment as a process which can and ought to be free from both measurement inaccuracy and human error. The results highlighted an underlying tension between the need to increase public understanding (of assessment error) and the need to retain public confidence (in our assessment systems). It was concluded that assessment agencies need to develop approaches to enhancing the public and professional understanding of assessment error, to counteract potentially misleading images from media reports.

Acknowledgements

The views expressed in this paper should not be taken to represent those of my employer, the Qualifications and Curriculum Authority. Nor have they been endorsed by any UK awarding body, regulatory authority or government department. I would like to thank Tim Oates and Ian Colwill for support in developing these ideas.

Notes

* Qualifications and Curriculum Authority, 83 Piccadilly, London W1J 8QA, UK. Email: [email protected]

1. The major stories emerged from the UK’s major assessment systems. National Curriculum tests, in English, mathematics and science, are administered annually to nearly all 7‐, 11‐ and 14‐year‐olds in England. For most of the UK, the principal pre‐university examination is the A level, and students tend to study for three or four A levels over a period of two years. A levels were unitized under the Curriculum 2000 reforms. Typically, for each subject studied, the first three (AS) units of Year 1 are ‘cashed in’ for an AS award; the second three (A2) units of Year 2 are ‘cashed in’—along with the corresponding AS units—for the full A level award. In Scotland, the principal pre‐university examination is the Higher.

2. The effect of this decision was to restrict the evidence base exclusively to articles written by official reporters (i.e., it excluded letters, opinion pieces, editorials, etc). In fact, many of the articles which concerned allegations against assessment agencies, and which were written by official reporters, also included quotations from agency representatives. In this sense, the agencies had quite a strong public voice.

3. Many articles were eliminated which were critical, and which had an assessment focus, but which were not directly concerned with specific allegations of error against an assessment agency. This included articles on: the burden of assessment on students and teachers; exam timetable clashes; stress associated with assessment; qualification uptake patterns; marker shortages; curriculum‐related concerns; late examination entries; costs of examining; accountability measures and performance targets; student or teacher cheating; general discussions of assessment, etc. The restricted focus also tended to exclude articles which concerned negative impacts from testing and results from surveys of public opinion. Concerns raised with proposals for new assessment systems were generally not included (unless they related directly to allegations of error within previous systems).

4. The DfES has ultimate responsibility for educational assessment policy decisions.

5. The QCA regulates the awarding bodies and has responsibility for National Curriculum testing.

6. Although there are over 100 awarding bodies in England, the three largest are responsible for the majority of the main pre‐university (e.g., A level) and school‐leaving (e.g., GCSE) examinations: the Assessment and Qualifications Alliance (AQA), Edexcel, and Oxford, Cambridge and RSA (OCR). All examining boards tend now to be referred to using the broader term, awarding body.

7. These agencies develop National Curriculum tests, under contract to the Qualifications and Curriculum Authority.

Additional information

Notes on contributors

Paul E. Newton*

* Qualifications and Curriculum Authority, 83 Piccadilly, London W1J 8QA, UK. Email: [email protected]