An Information-Based Approach to Identifying Rapid-Guessing Thresholds

ABSTRACT

The identification of rapid guessing is important to promote the validity of achievement test scores, particularly with low-stakes tests. Effective methods for identifying rapid guesses require reliable threshold methods that are also aligned with test taker behavior. Although several common threshold methods are based on rapid guessing response accuracy or visual inspection of response time distributions, this paper describes a new information-based approach to setting thresholds that does not share the limitations of other methods. A pair of information-based methods are introduced, and an empirical comparison study found the new methods to more reliably set thresholds than methods based on response accuracy or visual inspection.

Disclosure statement

No potential conflict of interest was reported by the author.

Notes

¹ By “low stakes” I am referring to the test taker’s perspective. That is, low stakes means that test takers are likely to perceive an absence of personal consequences associated with their test performance.

² Note the additional problem that without a bimodal distribution it is difficult to rule out the possibility that no rapid guessing had occurred on the item.

³ There have been two other types of threshold identification methods proposed that I did not consider in this study. Wise and Kong (2005) chose thresholds based on surface features of items such as their length and whether they contained tables or figures. Their criteria were arbitrarily chosen, however, and their method has not been used to set thresholds in subsequent research. Also, Schnipke and Scrams (1997) explored the use of two-state mixture models to set thresholds. This method cannot always estimate its model parameters, and it also has not been used in practice.

⁴ Criteria 4 and 5, which specify the presence of “sizeable” magnitudes of both information and accuracy discrimination, are necessarily a bit vague for two reasons. First, although rapid guesses should routinely exhibit correlations (information) near zero, rapid guessing accuracy can vary substantially across items. Second, response accuracy and correlations can also vary considerably across items, driven by an item’s difficulty and discrimination. Taken together, these reasons make it challenging to specify how much increase in correlations and accuracy should be expected to occur for a given item across the transition from rapid guessing to solution behavior.