From Products to Process: An Ecological Approach to Bias Detection

Abstract

Researchers from a range of disciplines (Faigley, 1995; Hamp-Lyons, 1994; Huot, 1996; Kirshner & Whitson, 1997; Lemke, 1995) have argued that to draw valid inferences from tests, more than the analysis of factors in isolation (e.g., items-test scores, rater consistency) should be investigated. The complex, interconnected, or "proximal"(Bronfenbrenner, 1994) relationships that form as a result of the processes and practices of testing activity should also become a focus of investigation. The study reported here is offered as an example of the application of an ecological approach in the investigation of the proximal relationship of raters (n = 12) and test takers (n = 423), formed during the trialling of a new science version of a high-stakes, topic-based language test. The study reveals a source of test bias that might otherwise have remained undetected by traditional methods, such as those that are common to differential item functioning studies. Systematically focusing on stakeholders' accounts of tests reveals more about how tests function, adds to the validity of the inferences drawn from test performance, and increases fairness.