The (Non)Impact of Differential Test Taker Engagement on Aggregated Scores

Abstract

Disengaged test taking tends to be most prevalent with low-stakes tests. This has led to questions about the validity of aggregated scores from large-scale international assessments such as PISA and TIMSS, as previous research has found a meaningful correlation between the mean engagement and mean performance of countries. The current study, using data from the computer-based version of the PISA-Based Test for Schools, examined the distortive effects of differential engagement on aggregated school-level scores. The results showed that, although there was considerable differential engagement among schools, the school means were highly stable due to two factors. First, any distortive effects of disengagement in a school were diluted by a high proportion of the students exhibiting no non-effortful behavior. Second, and most interestingly, disengagement produced both positive and negative distortion of individual student scores, which tended to cancel out much of the net distortive effect on the school’s mean.

Keywords:

• computer-based testing
• test motivation
• test-taking engagement

Notes

1 Rapid perfunctory responses were defined by Wise and Gao (2017) as rapid responses to a constructed response item that were very brief—shorter than 10% of the average length of responses given to the item by students.

2 For multiple-choice items, rapid guesses are not truly random. Wise and Kuhfeld (in press) showed that middle response categories tend to get chosen more often than the first or last options. This implies that the accuracy of rapid guesses depends on the position of the correct option. For example, Wise and Kuhfeld found that rapid guesses to 4-choice items were correct about 35% of the time when the correct option was “B” but correct only about 15% of the time when the correct option was “D.”

3 Calculated according to the binomial theorem.