The Effect of Small Group Discussion on Cutoff Scores During Standard Setting

Abstract

Standard setting methods, like the Bookmark procedure, are used to assist education experts in formulating performance standards. Small group discussion is meant to help these experts in setting more reliable and valid cutoff scores. This study is an analysis of 15 small group discussions during two standards setting trajectories and their effect on the cutoff scores on four performance levels for comprehensive reading and mathematics. Discussion decreased the variability of the cutoff scores among the expert panelists, but the direction of the adaptations varied among groups. Furthermore, also the duration and the content of the audio-taped discussions differed among groups. There was no relationship between the increase in agreement among the panelists and the duration of their discussions or their use of arguments concerning learning content. It was concluded that an increased consensus among panelists alone does not provide enough information on the reliability and validity of cutoff scores. Additional measures aimed at the content of group discussions have appeared to be necessary, since the use of content arguments in these discussions is not guaranteed.

Notes

¹The scale scores are arbitrary and the negative values do not reflect normative judgments.

²In the translation of the performance levels and the names of the students the alliteration was maintained. The original Dutch example students are: Marcel on the Minimum level, Fenny on the Basic (Dutch: “fundamenteel”) level, Sharissa on the Proficient (Dutch: “streef”) level, and Gerrit on the Advanced (Dutch: “gevorderd”) level.

³The personification of the performance levels seemed to have succeeded: during the group discussion, the panelists were found to refer to the names of the example students in terms of: My Michelle can do this; Lets shift to Benny now; or I’ve got two Arthurs in my classroom.

⁴The item numbers are not adjacent because they are ordered based on their difficulty within the texts.

FIGURE 2 Comparison of two groups discussing the cutoff score for the Proficient level of reading comprehension.

⁵The changes in the score ranges of within the groups can be deducted from Table 3: for every performance level, the range “before” discussion is subtracted from the range “after” discussion. For example, the change in range for group RC1 for the Minimum level is 7 – 0 (consensus) = 7 and the change in range for group M2 for the Minimum level is 11 – 2 = 9.