Accounting for unexpected test responses through examinees’ and their teachers’ explanations

Abstract

Researchers have developed indices to identify persons whose test results ‘misfit’ and are considered statistically ‘aberrant’ or ‘unexpected’ and whose measures are consequently potentially invalid, drawing the test’s validity into question. This study draws on interviews of pupils and their teachers, using a sample of 31 10‐year‐olds who were flagged as most ‘aberrant’ in a standardised mathematics test. The children’s and their teachers’ explanations were analysed and attributed: (i) to item‐, person‐ (self/other) and classroom‐levels; and ii) according to causal dimensions. Children’s and teachers’ explanations were mostly in agreement in relation to unexpected negative results and they included references to previously well‐cited sources of construct‐irrelevant variance (e.g. ineffective test‐taking strategies, careless mistakes) as well as construct‐relevant variance (e.g. misconceptions, weaknesses in particular topics). Findings of this exploratory study are discussed from a test validity and attribution theory perspective: we conclude that this approach offers grounds for multi‐level explanations of person misfit and that this qualitative research approach to unexpected responses is worthy of more attention.

Acknowledgements

The authors are grateful for the financial support of the ESRC (PTA‐030‐2004‐00072 and PTA‐026‐27‐1269), the A.G. Leventis Foundation and the University of Manchester.

Notes

1. The term ‘statistically unexpected response’ refers to responses in the test that have been signalled by the measurement model (i.e. Rasch) as unexpected. There are two types of ‘statistically’ unexpected responses: unexpectedly correct, right answers to relatively ‘difficult’ questions, the ability of the person is lower than the difficulty of the item; and unexpectedly wrong, wrong answers to relatively ‘easy’ questions, the ability of the person is higher than the difficulty of the item. These wrong responses to relatively easy questions or correct responses to relatively difficult questions produce large residuals responsible for their large fit‐statistic values. A response pattern would be regarded as ‘aberrant’ if it involved a significant number of unexpectedly correct and/or unexpectedly wrong responses. We refer to these responses as statistically unexpected because these responses have been signalled as ‘unusual’ or ‘unexpected’ by a statistical model (i.e. Rasch model).