Qualification users’ perceptions and experiences of assessment reliability

Abstract

This paper presents the findings of a study designed to explore qualification users’ perceptions and experiences of reliability in the context of national assessment outcomes in England. The study consisted of 17 focus groups conducted across six sectors of qualification users: students, teachers, trainee teachers, job-seekers, employers and employees. Each qualification user group discussed a series of issues concerned with the reliability of grades awarded at the completion of compulsory (age 16) and post-compulsory (age 18) education. Findings suggested that there were some notable differences between the participating groups in relation to their experiences of assessment reliability and their inclinations to trust the reliability of assessment outcomes. For example, teachers shared a number of experiences of suspected and actual error in the production of students’ grades, and of securing subsequent grade changes for their students. In the light of these experiences, teachers were disinclined to accept at face value the reliability of assessment outcomes. Among other groups, assessment outcomes were generally regarded as trustworthy and reliable measures of academic ability. However, without access to information about assessment reliability, participants were unaware that reliability can be undermined by random measurement error. Participants discussed measurement errors in relation to student well-being, facets of the test and the test situation and environment, but such errors were deemed to be an inevitable part of test-taking and as having limited impact on the reliability of assessment outcomes. It is concluded that efforts to enhance public understanding of assessment reliability, therefore, may be complicated by tensions between traditional, specialist conceptions of assessment reliability and what is perceived as relevant and useful by qualification users.

Keywords:

• assessment
• qualifications
• reliability
• public perceptions

Acknowledgments

I am grateful to the Ofqual for funding the first wave of data collection and analysis as part of the ‘Reliability of Results’ research programme. I would also like to thank the editors, Paul Black and Jo-Anne Baird, and the peer reviewers for their helpful comments.