The Impact of Setting Scoring Expectations on Rater Scoring Rates and Accuracy

ABSTRACT

Efficient constructed response (CR) scoring requires both accuracy and speed from human raters. This study was designed to determine if setting scoring rate expectations would encourage raters to score at a faster pace, and if so, if there would be differential effects on scoring accuracy for raters who score at different rates. Three rater groups (slow, medium, and fast) and two conditions (informed and uninformed) were used. In both conditions, raters were given identical scoring directions, but only the informed groups were given an expected scoring rate. Results indicated no significant differences across the two conditions. However, there were significant increases in scoring rates for medium and slow raters compared to their previous operational rates, regardless of whether they were in the informed or uninformed condition. Results also showed there were no significant effects on rater accuracy for either of the two conditions or for any of the rater groups.

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes

¹ Some raters scored the incorrect set of responses on Day 1 or Day 2 and were dropped from the analyses on those days. One rater was also dropped from Day 1 because the rater scored only a few essays that day.