Evaluation fidelity: An examination of criterion‐based assessment and rater training in the speech communication classroom

Abstract

The purpose of this study was to further investigate the effectiveness of speech evaluation procedures and measures. Specifically, this study examines the relationship between rater training and evaluation fidelity— a concept which focuses on the shared understanding of meaning between speech raters and students in terms of what is expected and what is performed. To do so, two studies were conducted. Study one hypothesized that evaluation fidelity would be associated with rater training such that inter‐rater (instructor‐instructor) reliability would be higher after training. The findings suggest that participants in Study one increased their evaluation fidelity in terms of total scores. The variance and frequency reports confirmed that the range of scores decreased (higher evaluation fidelity) after training. Study two hypothesized that instructors who provided their students with training in using the “Criteria for Evaluating Speeches” form would achieve higher evaluation fidelity than those instructors who do not. Results support this hypothesis and reveal higher evaluation fidelity between instructors and students in the group where students received a copy and explanation of the criterion‐based assessment tool.