Automated Scoring of Short-Answer Reading Items: Implications for Constructs

Abstract

This article examines how the use of automated scoring procedures for short-answer reading tasks can affect the constructs being assessed. In particular, it highlights ways in which the development of scoring algorithms intended to apply the criteria used by human raters can lead test developers to reexamine and even refine the constructs they wish to assess. The article also points out examples of ways in which attempting to replicate human rating behavior while avoiding incidental construct alteration can pinpoint areas of the automated scoring process requiring further development. The examples discussed in this article illustrate the central point that construct definitions should always guide the development of scoring algorithms while the process of developing and refining such algorithms requires more rigorous construct definitions and can potentially push us to refine our constructs.

ACKNOWLEDGMENTS

An earlier version of this article was presented at the 24th Language Testing Research Colloquium in Hong Kong and is based on work done while the authors were graduate students at the University of California, Los Angeles. We thank Michael J. Pan for answering several important technical questions about the WebLAS automated scoring system.

Notes

¹We consider short-answer reading tasks to be one type of constructed response task; more specifically, they are tasks whose expected responses require a limited amount of language production (i.e., responses ranging from one word to one sentence in length; Bachman & Palmer, 1996).

²By construct representation, we mean the extent to which the construct(s) of interest is/are observable in an assessment, by means of both the tasks (which should elicit responses providing evidence of the construct on the part of test takers) and the scoring method (which should provide a systematic means of deciding the extent to which evidence of the construct is displayed in test takers' responses).

³We are indebted to an anonymous reviewer for raising this issue.