An Examination of Individual Ability Estimation and Classification Accuracy Under Rapid Guessing Misidentifications

ABSTRACT

To mitigate the deleterious effects of rapid guessing (RG) on ability estimates, several rescoring procedures have been proposed. Underlying many of these procedures is the assumption that RG is accurately identified. At present, there have been minimal investigations examining the utility of rescoring approaches when RG is misclassified, and individual scores are reported. To address this limitation, the present simulation study investigates the effect of RG misclassifications on individual examinee ability estimate bias and classification accuracy when using effort-moderated (EM) scoring. This objective is accomplished by manipulating simulee ability level, RG rate, as well as misclassification type and percentage. Results showed that EM scoring significantly improved ability inferences for examinees engaging in RG; however, the effectiveness of this approach was largely dependent on misclassification type. Specifically, across ability levels, bias tended to be on average lower when falsely classifying effortful responses as RG. Although EM scoring improved bias, it was susceptible to elevated false-positive classifications of ability under high RG.

Disclosure Statement

No potential conflict of interest was reported by the author(s).

Notes

¹ This behavior is typically represented by short response times that are reflective of examinees not having adequate time to read all components of the item and solve the problem. However, it is possible that examinees can also engage in noneffortful responding by spending a significant amount of time on an item (i.e., slow noneffortful responding). Thus, rapid guessing may not encapsulate all forms of noneffortful responding.

² Recent extensions of these weighting schemes have been proposed from a Bayesian framework (Maeda & Zhang, 2020).

³ Further, to date, there has been limited research examining the effect of RG on individual ability estimates. Rios and Soland (2021) previously examined low effort examinees as a subgroup, but little consideration has been given to contexts where individual inferences are made.

⁴ In practice, some testing programs have changed from calibrating data from the 3PL to the 2PL model because of the former’s difficulty in estimating the pseudo-guessing parameter and the strong assumption that response disturbances (e.g., guessing) are equivalent across all examinees, which may be untenable in practice (Wise & Kingsbury, 2016).

⁵ Wise and Kingsbury (2016) also noted that RG can occur idiosyncratically at the examinee-level. However, the simulation approach taken in this manuscript has been shown to be associated with nearly identical degrees of bias compared to idiosyncratic RG (see Rios & Soland, 2021). The rationale for the similar effects observed across RG patterns is that both lead to simulees engaging in RG on items that vary in difficulty, and thus, the biasing effect on ability estimates is comparable.

⁶ Although it is possible that both false-positive and false-negative errors can be simultaneously made when classifying RG responses, it is likely that this would largely lead to a cancellation effect. Simulation research is needed to support this hypothesis.

⁷ To increase the generalizability of findings, misclassification rates for a cut-point of −0.84 logits were examined, similar to Rios and Deng (2021). Results of this analysis are provided in Appendix B of the supplemental file.