Evaluating Panelists' Standard Setting Perceptions in a Developing Nation

Abstract

Considerable research about standard setting has revolved around a U.S.-centric policy context. That is, over the past decade, conclusions about thought processes and the interaction of education policy and panelists' judgments have been based on assumptions of comparable policy settings. However, whether these assumptions generalize to other education contexts is to some extent unknown. This study addresses what standard setting panelists in a developing nation think about when making item performance estimates for students in various performance categories. Panelists were asked to respond to a set of questions at various stages of the standard setting workshop. The results suggested that panelists gained experience about standard setting procedure, understanding descriptions of the performance level categories, and making more consistent decisions on the test items. Most of the panelists were not influenced by the item p-value and impact data while making Round 2 ratings. However, the panelists did mention that the feedback data helped them to understand better the four performance level categories and provided a reality check of their Round 1 item performance estimates. Salient factors for panelists in this study are compared with findings from other research in the area.

Keywords:

• developing country
• panelists’ perceptions
• standard setting