The Effect of Data Format on Integration of Performance Data Into Angoff Judgments

Abstract

This study investigated the extent to which the performance data format impacted data use in Angoff standard setting exercises. Judges from two standard settings (a total of five panels) were randomly assigned to one of two groups. The full-data group received two types of data: (1) the proportion of examinees selecting each option and (2) plots showing the proportion of examinees selecting the correct answer by deciles defined by total test score. The options-only group received only the option data. Results indicated that judgments in the full-data group were in substantially closer alignment with the empirical data than those in the options-only group. This suggests that either the decile data alone or the combination of both pieces of data leads to a greater reliance on the data. The results are discussed from the perspective of the validity/credibility of the resulting cut scores.

Keywords:

• Angoff procedure, credentialing, high-stakes testing, performance data, standard setting, validity